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Theme 


The  complexity  of  the  modern  battle  scenario  is  demanding  better  situational  awareness  for  the  pilot/crew.  The  advent  of  laser 
weapons  capable  of  blinding  a  pilot  or  crew  member  requires  that  such  radiation  should  not  reach  the  pilot.  This  dictates  that 
windowless  or  severely  restricted  visibility  cockpit  concepts  be  used.  Emerging  technologies,  properly  applied,  offer  potential 
solutions  to  answer  these  conflicting  issues. 

The  present  interface  between  the  pilot/crew  and  the  aircraft  is  evolving  to  more  sophisticated  displays  represented  on  a  variety 
of  media  including  CRTs,  flat  panels,  and  helmet  mounted  displays  augmented  by  multi-function  switches  and  voice.  The 
application  of  these  technologies  presents  both  opportunities  and  special  requirements.  These  requirements  include  the 
development  and  integration  of  a  variety  of  concepts  to  enhance  the  situational  awareness  while  reducing  the  laser  threat. 

The  exploration  of  these  concepts,  their  integration  in  various  combinations,  and  their  potential  to  enhance  mission  capability 
was  the  central  theme  of  this  Symposium. 


Theme 


La  complexity  croissante  des  scenarios  de  combat  modernes  exige  une  meilleure  perception  de  la  situation  pour  le  pilote  ou 
('equipage.  L’apparition  d'armes  laser  capables  d’aveugler  le  pilote  ou  un  autre  membre  de  1’equipage  exige  que  de  tels 
rayonnements  n’atteignent  pas  le  pilote.  Ceci  impose  le  recours  a  des  concepts  d'habitacles  sans  vitre  ou  a  visibility  tres  reduite. 
Les  technologies  recentes,  si  elles  sont  appliquees  correctement  peuvent  fournir  des  solutions  pour  repondre  a  ces  questions 
contradictoires. 

Les  equipements  d’interface  entre  l’equipage  et  l’avion  evoluent  vers  des  systemes  complets  plus  perfectionnes  avec  affichage  de 
symboles,  presente  sur  divers  equipements  tels  que  les  ecrans  cathodiques,  les  ecrans  plats  et  les  visualisations  montes  sur  le 
casque  augmentes  de  selecteurs  multifonctions  et  de  commandes  vocales.  Si  la  mise  en  oeuvre  de  ces  technologies  presente  des 
opportunity,  elle  pose  aussi  des  besoins  specifiques.  Ces  besoins  comprennent  le  developpement  et  1’integration  de  differents 
concepts  pour  rehausser  la  perception  de  la  situation  tout  en  reduisant  la  menace  laser. 

L'examen  de  ces  concepts,  leur  integration  selon  differentes  configurations,  et  leur  potentiel  pour  ('amelioration  de  la  capacite 
operationnelle,  constituaient  le  theme  de  ce  symposium. 
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TECHNICAL  EVALUATION  REPORT 


by 


Dr.  Geoffrey  H.  Hunt 
lately  of  the  Defence  Research  Agency 
RAE 

Famborough,  Hants,  GU14  6TD 
United  Kingdom. 


INTRODUCTION 

A  Symposium  of  the  AGARD  Avionics  Panel  was  held 
in  Madrid,  Spain,  on  May  18  to  May  21  1992.  The 
subject  of  the  symposium  was  "Advanced  Aircraft 
Interfaces:  The  Machine  Side  of  the  Man  Machine 
Interface".  The  Programme  Chairman  for  the  meeting  was 
Mr.  William  E.  Howell  of  NASA. 


GENERAL  DESCRIPTION 

The  programme  consisted  of  thirty-two  papers,  divided  into 
seven  sessions.  Preceding  these  was  a  keynote  address  which 
provided  an  excellent  introduction  to  the  later,  more  detailed, 
papers.  Discussion  was  invited  after  each  paper,  but  no 
period  was  allocated  for  general  or  round-table  discussions. 


THEME  OF  THE  SYMPOSIUM 


TECHNICAL  EVALUATION 


The  complexity  of  the  modem  battle  scenario  is 
demanding  more  of  the  pilot/crew  in  the  aircraft  and  their 
need  for  better  situational  awareness  is  progressively 
increasing.  Furthermore,  the  advent  of  laser  weapons 
capable  of  blinding  a  pilot  or  crew  member  is  demanding 
that  concepts  be  used  which  preclude  such  radiation 
reaching  the  pilot.  This  requirement  dictates  that 
windowless  or  severely  restricted  visibility  cockpit 
concepts  be  used;  however  the  need  still  exists  for 
complete  visibility  in  all  directions  and  wavelengths,  to 
avoid  or  engage  hostile  forces.  Emerging  technologies, 
properly  applied,  offer  potential  solutions  to  these  issues 
including  retrofit  of  existing  weapons  systems  to  meet  the 
changing  requirements. 

The  present  interface  between  the  pilot/crew  and  the 
aircraft  is  evolving  to  more  sophisticated  full  colour, 
abstract  displays  presented  on  a  variety  of  media 
including  CRT’s,  flat  panels  and  helmet-mounted 
displays.  The  pilot’s  inputs  will  still  include  standard 
buttons  and  switches  but  will  be  augmented  by  multi¬ 
function  switches  and  possibly  voice.  The  application  of 
this  technology  presents  both  opportunities  and  special 
requirements.  These  requirements  include  the 
development  and  integration  of  a  variety  of  concepts  such 
as  CRT  Displays,  Flat  Panel  Colour  Displays,  Helmet 
Mounted  Displays  (including  helmet  optical  geometry). 
Head  Up  Displays.  Night  Vision  Goggles.  Integrated 
Displays  (Visual.  IR.  Computer-generated  symbology). 
Voice  I/O.  Multi-Function  Display  and  Input  Devices, 
and  Target  Acquisition  and  Aiming  Devices. 

The  exploration  of  these  concepts,  their  integration  in 
various  combinations,  and  their  potential  to  enhance 
mission  capability  were  the  central  theme  of  the 
Symposium. 


Session  I  -  Defining  Concepts  and  Design  Issues 

In  this  first  session  it  would  have  been  useful  had  a 
framework  been  set  for  the  subsequent  sessions.  In  fact  two 
of  the  papers  provided  this  in  par  t.  C.G.Burge  (Paper  3)  gave 
a  top-down  analysis  of  the  problems  in  military  aircraft  man- 
machine  interfaces  and  listed  nine  categories  which  required 
research,  and  S.P.Williams  (Paper  2)  gave  a  series  of 
examples  of  cockpit  equipments  which  are  being  investigated 
for  use  in  future  civil  transport  aircraft  However,  neither 
author  made  clear  the  relationship  between  the  interface 
design  requirements  and  the  characteristics  of  the  avionics  or 
weapons  systems  with  which  they  were  to  be  integrated. 

Professor  Bosman  (Paper  1)  described  the  characteristics  of 
the  human  eye  and  gave  some  particularly  pertinent 
observations  on  the  scan  patterns  when  reading  text  or 
numerals  as  compared  with  those  when  looking  at  complex 
scenes;  these  are  of  obvious  importance  when  considering 
display  layouts  for  use  at  times  of  high  workload.  It  would 
have  been  interesting  if  the  information  given  on 
monochrome  displays  had  been  extended  to  colour  since 
almost  all  displays  in  modern  aircraft  are  now  multi-coloured. 

Paper  4  by  J. Struck  was  a  description  of  alternative  computer 
system  arrangements  for  generation  of  information  in  a 
graphical  form,  an  important  part  of  the  design  of  display 
interfaces.  It  did  not  provide  any  information  about  the 
benefits  or  trade-offs  of  the  alternative  designs. 

Paper  5.  which  was  presented  in  the  absence  of  the  authors 
by  L.Dopping-Hepenstal.  reported  on  a  questionnaire  survey 
of  selected  British  Aerospace  pilots  concerning  the 
automation  of  airborne  systems.  Ten  distinct  categories  of 
pilot-machine  interface  were  defined,  ranging  from  the  pilot 
performing  the  entire  activity  to  the  system  autonomously 
performing  the  action.  Opinions  generally  favoured  increased 
use  of  automation,  with  pilots  wanting  to  be  informed  of  the 
effects  of  faults  upon  the  capability  of  the  aircraft  rather  than 
the  technical  diagnosis  of  specific  types  of  fault.  Direct 
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intervention  into  the  operation  of  the  flight  control  system 
was  declared  unacceptable. 


Session  II  •  Maintenance  for  Advanced  Cockpit 
Systems 

Session  II  consisted  of  only  two  papers.  The  subject  of 
the  session  is  an  important  one  since  it  is  clearly 
necessary  to  ensure  that  cockpit  equipments  have 
adequate  availability,  reliability  and  integrity  compatible 
with  their  role  as  the  link  between  the  aircrew  and  all  the 
aircraft  systems.  However  it  is  not  apparent  that  there 
need  be  any  difference  between  the  maintenance  of 
cockpit  systems,  which  are  very  largely  computer-based, 
and  that  of  other  computer-based  onboard  equipment. 
This  was  confirmed  by  the  two  presented  papers. 

Paper  6.  presented  by  J.A. Collins,  was  particularly 
interesting  in  describing  the  evolution  of  a  new  method 
of  improving  maintenance  methodology  by  implanting 
devices  into  avionics  equipment  that  would  measure  and 
record  physical  parameters  that  were  judged  to  affect  the 
failure  rates  of  the  equipment  into  which  they  had  been 
implanted,  for  example  temperature  and  shock.  Using  the 
outputs  from  these  sensors,  together  with  a  model  of  the 
predicted  failure  pattern,  it  is  hoped  that  many  B1T- 
detected  failures  which  cause  system  cut-out  can  be 
confirmed  as  soft  failures  and  hence  allow  system  restart. 

Paper  7,  presented  by  W.Wurster,  described  a 
conventional  computer-aided  maintenance  system  which 
was  claimed  to  be  particularly  useful  during  the 
development  phase. 

Session  III  •  Panoramic  and  Virtual  Cockpits 

Advances  in  both  computer  and  display  technology  are 
bringing  closer  to  reality  the  concept  of  the  "Virtual 
World"  in  which  a  human  operator  using  all  of  his  input 
and  output  mechanisms  can  believe  himself  to  be  in  a 
real  world  which  responds  correctly  to  any  action  which 
he  cares  to  make  even  though  this  world  has  actually 
been  created  artificially.  As  applied  to  aircraft.  Virtual 
Cockpits  are  usually  taken  to  mean  cockpits  in  which  the 
pilot’s  visual  scene,  including  both  the  outside  world  and 
the  instrument  panel,  are  created  and  displayed 
artificially,  while  aural  and  touch  sensors  are  also  fed 
from  computers  through  suitable  transducers.  Paper  8. 
presented  by  W.L.Martin,  described  a  US  AF  development 
programme  in  this  area.  The  absence  of  any  detailed 
engineering  data,  such  as  the  resolution  obtainable  from 
the  displays,  or  any  results  of  trials,  significantly  reduced 
the  value  of  the  paper  and  made  it  impossible  to  assess 
whether  the  concept  is  likely  to  become  an  effective  and 
useful  interface  in  the  forseeable  future. 

Paper  9,  presented  by  D.G. Hopper,  described  another 
USAF  programme,  this  one  concerned  with  simulator 
trials  of  a  very  large  centrally-mounted  head-down 
display.  The  display  was  used  to  provide  better 
situational  awareness  for  a  pilot  in  air-to-air  combat  than 
that  used  in  the  standard  F-15.  Significantly  improved 
kill  ratios  were  claimed. 

Paper  10.  by  P.Larroque  and  R. Joannes,  also  described  a 
development  programme  leading  up  to  simulator  trials,  in 
this  case  of  low-level  penetration  flights  in  poor  weather 
or  at  night.  The  technique  used  was  to  create  a  synthetic 


display  of  the  outide  world,  based  on  a  data  base  created 
from  reconnaissance  and  map  information.  The  results  from 
the  first  series  of  trials  were  encouraging,  with  positive  pilot 
acceptance,  and  this  will  lead  to  further  phases  which  will 
include  studies  of  integrity  and  of  combining  the  images  from 
onboard  sensors  with  the  synthetically-created  data. 


Session  IV  -  Helmet  Mounted  Displays 

Session  IV  was  the  biggest  of  the  Symposium  with  7  papers, 
although  one  (Paper  14)  was  not  presented.  This  reflects  the 
continuing  interest  in  helmet-mounted  displays  as  an  R&D 
topic  in  spite  of  the  persistant  difficulty  in  finding 
engineering  solutions  to  the  intrinsic  problems  of  mounting 
wide-angle,  high-resolution  displays  on  a  helmet  in  such  a 
way  that  the  overall  helmet  design  is  compatible  with  all  the 
safety  and  usability  criteria  for  military  aircraft  use. 

Tlie  use  of  diffractive  optics  is  one  way  of  finding  an 
acceptable  solution  to  the  problem  of  providing  wide  field  of 
view  displays  close  to  the  pilot's  eyes.  Paper  11,  presented 
by  G.  de  Vos,  gave  an  account  of  the  design  trade-offs  and 
explained  that  the  need  to  reduce  weight  required  that  relay 
optical  elements  should  be  minimised  and  hence  that  the 
holographic  elements  should  not  be  far  off-axis.  A  novel 
design  of  visor  hologram  was  described  which  reduces 
chromatic  aberration. 

Paper  12.  given  by  S.  Williams,  provided  guidelines  on  the 
possible  application  of  stereoscopic  helmet-mounted  displays 
to  a  number  of  different  types  of  aircraft  operation.  The 
most  clear-cut  advantages  were  found  in  the  precision  hover 
of  rotorcraft;  for  other  operations  such  as  a  tracking  task  and 
a  curved  approach  to  landing,  the  gain  in  performance  was 
found  to  be  significantly  less. 

J.P.Cursolle  gave  paper  13  on  the  subject  of  applications  of 
helmet-mounted  displays  to  the  operation  of  combat  aircraft 
at  night  and  in  all  weather.  The  pilot’s  need  for  "contact" 
flying  was  explained  as  well  as  the  essential  characteristics 
of  a  helmet-mounted  dislay  appropriate  for  this  type  of 
operation. 

Paper  15.  which  was  presented  by  A.Karavis  and  T.Southam, 
described  the  MOD(UK)  programme  which  is  attempting  to 
develop  a  fully-engineered  helmet  which  will  meet  a  design 
specification  appropriate  to  a  high  performance  combat 
aircraft.  The  helmet  will  cany  a  binocular  wide-angle  display 
and  a  pointing  pickoff,  as  well  as  the  normal  fittings  such  as 
facemask.  The  detailed  specification  was  presented,  which 
has  been  derived  from  earlier  MOD  experience  of  flying  non- 
integrated  helmet-mounted  displays. 

Paper  1 6,  by  J.P.Foley.  descibed  optical  techniques  which  can 
be  used  to  provide  protection  to  the  pilot's  eyes  against 
damage  resulting  from  flashes  and  lasers.  A  good  analysis 
was  given  of  the  available  technologies,  both  passive  and 
active. 

W.E. Howell  presented  Paper  17  which  gave  an  assessment  of 
the  use  of  synthetic  vision  for  the  approach  and  landing  of 
civil  transport  aircraft.  He  described  trials  carried  out  by 
NASA  in  a  Boeing  737  aircraft  in  which  the  landing 
accuracy  was  found  to  be  approximately  equivalent  to  that 
obtained  with  normal  vision.  Although  the  experiments  were 
principally  directed  to  the  use  of  synthetic  vision  on  future 
spacecraft,  the  results  have  significance  for  military  aircraft 
as  well. 


Session  V  •  Voice  Technology 


earlier  sessions. 


Voice  forms  a  natural  means  of  man-to-man 
communication  which  has  always  been  extensively  used 
in  military  aircraft,  and  experiments  have  been  conducted 
for  several  years  to  develop  the  technology  for  use  in 
both  man-machine  and  machine-man  roles.  Session  V 
comprised  five  papers  which  gave  a  reasonable  overview 
of  the  curret  state-of-the-art  in  this  area. 

Paper  18  by  Professor  N.Ince  gave  a  good  introductory 
review  of  speech  coding,  compression  and  recognition,  of 
which  only  synthesis  and  recognition  can  really  be 
considered  man-machine  interface  techniques.  For 
synthesis,  a  high  probability  of  successful  operation  is 
now  available  from  relatively  cheap  off-the-shelf 
equipment.  Voice  recognition  is  much  more  difficult,  and 
hence  most  systems  have  to  accept  severe  limitations 
such  as  reduced  vocabulary  and  speaker  dependance. 
Even  with  these  limitations  there  is  less  than  100% 
probability  of  correct  recognition  which  implies  that  for 
most  applications  supplementary  verification  will  be 
necessary. 

Paper  19  by  F.Hollcvoel  concentrated  on  voice 
recognition  and  gave  a  description  of  the  recognition 
process  in  a  typical  system,  it  provided  a  useful 
reinforcement  of  many  of  the  points  made  in  Paper  18. 

Paper  20,  presented  by  B.Barbier.  gave  a  description  of 
some  preliminary  experiments  to  study  possibilities  for 
future  cockpit  layouts.  They  explored  the  possibility  of 
using  multiple  inputs  from  the  pilot  such  as  eye 
movement,  voice  input,  and  hand  and  head  position 
transducers.  Unfortunately,  the  results  of  the  experiments 
were  not  given. 

Paper  21  was  given  by  M.M.Taylor.  and  provided  a 
description  of  the  psychological  background  to 
communication  including  the  use  of  voice  as  the 
communication  medium  in  a  man-machine  control  loop. 
The  author  developed  a  rather  philosophical  discussion  of 
the  principles  of  man-machine  interface  and  the  sharing 
of  control  between  pilot  and  aircraft  system. 

C.Gulli  presented  paper  22  which  gave  an  account  of  an 
experimental  programme  on  voice  recognition  exploring 
the  effects  on  recognition  performance  of  influences  such 
as  cockpit  noise,  g-loading  and  oxygen  mask. 
Experiments  carried  out  in  a  centrifuge  showed  that  the 
recognition  performance  should  not  be  affected  by  g 
loads  if  proper  compensation  is  provided:  this  is 
encouraging  as  acceleration  effects  are  often  quoted  as  a 
major  problem.  A  good  presentation  of  the  experimental 
results  was  given.  The  author  expressed  optimism  about 
future  possibilities  while  emphasising  the  necessity  for 
more  research. 


Session  VI  -  System  Design  Concepts  and  Tools 

Session  VI  was  principally  concerned  with  the 
methodology  of  the  design  and  development  of  the  man 
machine  interface,  but  inevitably  several  of  the  papers 
also  discussed  some  of  the  broader  issues  concerning  the 
relationship  between  the  interface  and  the  total 
aircraft/aircrew  system.  Perhaps  because  of  these  wider 
systems  implications,  the  discussions  following  each 
paper  were  noticeably  more  lively  than  they  had  been  in 


Paper  23  by  F.Amiogida  discussed  the  reduction  of  pilot 
workload  in  critical  phases  of  a  combat  mission  by  the 
mechanism  of  transferring  work  to  earlier  phases  of  the 
(light.  He  advocated  more  use  of  airborne  mission  planning 
and  its  proper  correlation  with  pre-flight  mission  planning. 
This  was  an  important  paper  in  recognising  that  ma.  -machine 
interface  problems  are  profoundly  affeetd  by  the  design  of 
the  complete  system. 

Paper  24  by  E.Lovesey  was  largely  devoted  to  a  historical 
perspective  on  cockpit  design  problems.  Its  main  interest  was 
that  it  was  one  of  the  very  few  papers  in  the  symposium 
which  even  mentioned  the  aircrew  problems  in  large 
multicrew  aircraft  and  the  need  to  consider  the  internal 
distribution  of  inlormation  and  of  tasking. 

Paper  25  by  C.R.Ovenden  described  some  investigations  of 
cockpit  warning  systems  for  use  in  civil  transport  aircraft,  but 
having  implications  for  military  aircraft  as  well.  The 
intentii  n  is  to  develop  diagnostic  systems  which  would 
monitor  the  health  of  aircraft  equipments  and  systems  and  be 
able  to  advise  the  crew  on  collective  action  which  should  be 
taken  in  advance  of  total  failures.  This  very  desirable 
objective  can  only  be  achieved  by  the  development  of  models 
of  system  behaviour  which  take  into  account  failure  trends 
and  hence  appears  likely  to  be  most  applicable  to  mechanical 
systems  such  as  engines.  No  information  was  provided  on 
how  far  the  work  has  progressed. 

Paper  26  was  presented  by  L.Dopping-Heponstal  and 
described  the  design  process  adopted  for  the  cockpit  and 
systems  of  the  Eurofighter.  The  process  featured  a  very 
structured  approach  in  which  the  information  flow  and  the 
crew  workload  during  a  small  number  of  well-defined 
missions  were  analysed  in  detail.  The  methodology  provided 
for  a  lot  of  analysis  and  work-station  testing  before 
commencing  mock-up  trials. 

C. Weber  (Paper  27)  described  a  cockpit  design  and 
development  tool  which  had  been  used  in  the  development  of 
the  "Tiger"  helicopter.  By  contrast  with  the  methc  1  logy 
described  in  paper  26.  it  appeared  that  this  was  based  on  the 
use  of  a  mock-up  right  from  the  start  of  the  design  process. 
A  good  description  was  given  of  the  experimental  results 
obtained,  which  included  both  performance  analysis  and 
aircrew  opinion  ratings. 

Paper  28  by  M.P.Kibbe  was  a  description  of  the  use  of 
automatic  target  recognition  systems  in  aircraft  and  their 
acceptability  by  aircrew.  It  was  stated  that,  for  various 
reasons,  automatic  systems  require  aircrew  back-up,  and  that 
to  make  this  effective  the  new  need  to  have  trust  in  the 
system  and  to  be  infoimed  of  the  source  of  the  recognition 
data.  The  experiments  for  both  identification  and  recognition 
of  ships  were  described:  the  results  were  not  clear-cut  and 
depend  on  the  reliability  of  the  perform*  of  the  automatic 
recognise  rs. 

Session  VII  -  Device  Technologies 

It  was  rather  surprising  that  only  four  papers  were  presented 
in  the  Device  Technologies  Session,  of  which  only  the  first 
two  could  reasonably  1*  described  as  papers  about  devices. 
This  is  perhaps  a  reflection  of  a  growing  awareness  that  most 
of  the  main  challenges  in  the  man-machine  interface  are 
concerned  with  the  total  system  rather  than  the  interface 
devices  themselves.  Nevertheless  there  is  little  doubt  that  the 
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performance  of  interface  devices,  particularly  displays, 
are  currently  •>.  limiting  factor  in  overall  system 
performance,  and  l'...t  the  colour  CRT's  v.  \ch  are 
predominantly  used  have  many  undesirable  features. 

Both  Paper  29  (presented  oy  J.C. Wright)  and  Paper  30 
(presented  by  F.deLauzun)  described  colour  liquid-crystal 
displays.  Several  advantages  were  claimed  over  CRT's, 
including  weight,  depth,  reliability  and  ease  of 
maintenance.  In  spite  of  problems  in  obtaining  good 
luminous  efficiency,  paper  29  claimed  that  contrast  ratio 
in  bright  sunlight  was  better  than  for  CRT’s.  Possibilities 
for  the  future  included  fault-tolerant  and  autostereoscopic 
displays. 

Paper  31  by  C.K.Lam  described  the  possible  use  of 
intelligent  controllers  to  optimise  the  characteristics  of 
automatic  target  cuers.  The  target  model  is  carried  in  a 
target  correlator  and  fuzzy  logic  is  used  for  controlling 
filter  and  parameter  choices.  No  results  were  given. 

Paper  32  by  H.Hellmuth  provided  another  statement  of 
the  man-machine  interface  problem,  in  this  case  in  the 
context  of  helicopter  operation.  Equipment  becoming 
available  was  described,  and  the  use  of  liquid  crystal 
displays  was  advocated. 

General  Comments 

The  evaluation  of  each  of  the  sessions  given  above  does 
not  provide  an  overall  assessment  of  how  well  tne 
objectives  of  the  Symposium  were  met.  and  it  is  the 
purpose  of  this  section  of  the  report  to  provide  this 
additional  overview. 

It  should  first  be  noted  that  the  discussions,  limited 
though  they  were,  did  not  reveal  any  significant 
disagreements  on  the  nature  of  the  problems  facing  the 
designers  of  military  aircraft  cockpits  and  interfaces.  It 
was  generally  agreed  that  these  problems  were  primarily 
related  to  the  excessive  workload  which  was  experienced 
by  combat  aircrew  which  modem  avionics  systems  had 
done  little  to  alleviate  (and  some  would  claim  had 
actually  exacerbated).  There  was  also  a  general  consensus 
on  the  range  of  solutions  available  (or  likely  to  become 
available)  to  designers  and  on  the  methodology  required 
to  apply  them. 

It  was  also  quite  clear  that  the  topic  and  theme  of  the 
symposium  were  generally  regarded  as  very  pertinent, 
though  the  limitation  to  the  machine  side  of  the  man- 
machine  interface  was  regarded  by  some  as  a  rather 
artificial  constraint,  even  though  a  subsequent  AGARD 
symposium  was  scheduled  to  cover  the  other  side. 

The  emphasis  of  the  symposium  was  quite  clearly  on  two 
main  topics.  The  first  of  these  was  concerned  with  the 
overall  problem  of  aircrew  management  of  a  highly 
automated  avionics  system  and  the  interface  implications 
thereof.  There  seemed  to  be  a  lack  of  clear  understanding 
by  many  of  the  authors  of  the  distinction  between 
needing  better  interfaces  to  reduce  the  workload 
associated  with  current  types  of  avionics  system  and 
needing  new  interfaces  to  reflect  the  new  types  of  data 
flow  which  are  necessary  for  more  automated  systems 
which  are  themselves  intended  to  reduce  workload.  In 
this  context  it  was  very  surprising  that  no  papers  were 
given  on  the  interface  aspects  of  the  three  programmes 
specifically  intended  to  provide  workload-reducing 
backup  to  the  crews  of  single-seat  combat  aircraft,  viz  the 


USA's  Pilot  Associate,  France's  Copilote  Electronique  or  the 
UK's  Mission  Management  Aid.  It  should  also  be  remarked 
dial  the  crew  workload  in  a  combat  or  strike  aircraft  and  the 
optimum  design  of  the  corresponding  man-machine  interface 
are  highly  dependant  on  the  characteristics  of  the  weapons 
being  used,  yet  none  of  the  papers  discussed  the  trends  in 
weapons  systems  (e.g.  their  capability  for  target  seeking)  and 
the  implications  of  these  for  the  aircrew. 

The  second  main  topic  was  concerned  with  the  Visual 
element  of  the  interface,  i.e.  on  displays.  Of  the  32  papers, 
no  fewer  than  13  were  directly  addressed  to  this  topic.  Such 
heavy  emphasis  is  understandable  since  the  visual  channel  is 
the  one  which  has  the  ability  to  convey  to  the  aircrew  much 
more  information  than  any  of  the  others.  However  it  has  to 
be  said  that  other  important  interfaces,  particularly  those 
concerned  with  commands  from  the  crew  to  the  machine, 
were  hardly  mentioned. 

There  was  also  an  overemphasis  on  the  problems  of  the 
interface  in  single-seat  combat  aircraft  which,  although 
understandable  in  terms  of  the  magnitude  of  the  problems 
involved,  resulted  in  almost  total  neglect  of  muliicrew 
aircraft.  It  would  certainly  have  been  interesting  to  have  had 
one  or  two  papers  on  the  man  machine  interface  issues  in 
AEW  and  ASW  aircraft  and  helicopters. 

One  final  comment  should  be  made  as  part  of  the  technical 
evaluation.  Very  few  papers  gave  a  description  of  Research 
and  Development  programmes  whereas  these  should  surely 
form  the  core  of  any  AGARD  Symposium.  Too  many  papers 
were  concerned  with  concepts  and  possibilities,  and  although 
such  papers  are  valuable  in  their  own  way,  they  are  no 
substitute  for  the  hard  facts  which  are  contained  in  a  well- 
written  account  of  work  carried  out  in  an  R&D  Laboratory. 

Taken  as  a  whole,  the  31  papers  which  were  presented  at  the 
Symposium  gave  to  the  participants  a  good  picture  of  the 
way  in  which  this  important  technological  area  is  perceived 
by  the  experts  working  in  the  field.  No  major  technological 
breakthroughs  were  described,  but  rather  a  steady  advance  in 
several  areas  which,  taken  together,  show  some  possibilities 
for  overcoming  many  of  the  inherent  difficulties  of  the  man- 
machine  interface  in  aircraft.  Evaluated  against  the  theme  of 
the  v  ymposium,  the  objectives  were  very  largely  met  and  the 
proceedings  should  stand  as  a  useful  record  of  the  state-of- 
the-art  as  it  currently  exists  and  a  pointer  to  the  way  in  which 
the  interface  technology  is  likely  to  evolve  in  the  future. 


THE  ORGANISATION  OF  THE  SYMPOSIUM 

The  symposium  was  held  in  an  excellent  conference  hall, 
well  able  to  accomodate  all  the  135  participants,  and  the 
projection  facilities  and  simultaneous  translation  were  also 
very  good.  The  only  significant  cause  for  criticism  was  the 
fact  that  less  than  half  of  the  papers  presented  were  available 
in  written  form,  even  though  many  of  the  authors 
acknowledged  that  the  written  papers  contained  much 
material  that  was  not  included  in  the  oral  presentation.  The 
absense  of  pre-prints  which  could  be  studied  in  advance  may 
also  have  contributed  to  the  lack  of  wort'- while  discussion 
following  many  of  the  presentations. 
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WELCOME  ADDRESS 

by 

EXCMO  Snr. 

D.  GONZALO  RODRIGUEZ  MARTIN-GRANIZO 
ALMIRANTE 

JEFE  DEL  ESTADO  MAYOR  DE  LA  DEFENSA 
c/Vitruvio  N.  1 
28006  MADRID 
Spain 


Chairman  of  the  Avionics  Panel,  Symposium 
Participants,  Ladies  and  Gentlemen, 

As  Chief  of  Spanish  Defence  Staff  and  subsequently,  as 
Member  of  the  NATO  Military  Committee,  I  feel 
particularly  delighted  to  welcome  you  to  my  country  and 
chair  this  opening  ceremony  of  the  scientific  sessions 
that  you  will  hold  throughout  the  week,  organized  by  this 
important  Panel  of  AGARD. 

This  is  the  second  time  that  I  have  had  the  honor  to  open 
an  AGARD  Symposium.  My  first  opportunity  was  in 
Seville,  about  one  year  ago,  when  the  Flight  Mechanics 
Panel  of  AGARD  held  a  Symposium. 

This  year,  AGARD  (founded  in  1952  by  that 
unforgettable  Scientist,  Professor  Theodor  VON 
KARMAN),  celebrates  its  fortieth  anniversary. 
Throughout  these  years  AGARD  has  been  and  continues 
to  be  an  extraordinary  forum  for  international  cooperation 
in  which  scientists  of  NATO  Countries  can  exchange 
experiences  and  work  together  in  scientific  meetings  on 
subjects  of  common  interest  in  the  aerospace  field  and 
related  disciplines. 

We  can  say  that  AGARD  is  today  one  of  the  most  useful 
NATO  Agencies.  Spanish  participation  in  AGARD 
activities  has  follow  an  increasing  role.  The  number  of 
Spanish  professionals  taking  an  active  part  in  these 
activities  is  growing,  either  as  Panel  Members,  or 
Members  in  Working  Groups,  or  Authors.  At  the  same 
time  we  are  increasing  the  rate  at  which  we  host 
AGARD  activities.  In  1992.  for  example,  we  shall  hold 
here,  besides  this  Symposium,  two  Lecture  Series, 
organized  by  the  Aerospace  Medicine  Panel  and 
Guidance  Control  Panel.  This  does  not  consider  the 
various  consultants  missions  and  Working  Group 
Meetings  affiliated  with  Spain. 

I  want  to  take  advantage  of  this  occasion  to  thank 
General  BAUTISTA  for  his  outstanding  work  for 
AGARD  for  the  4  past  years.  Up  until  his  recent 
retirement  he  served  as  s  Spanish  National  Delegate  to 
AGARD.  At  the  same  time  I  would  like  to  congratulate 
General  GUITART,  nominated  as  new  National  Delegate, 
who  I  know,  will  continue  this  fostering  cooperation 
mission,  with  enthusiastic  dedication. 

Changes  in  the  World's  political  situation,  especially  the 
changes  in  the  former  Soviet  Union,  the  abolishment  of 


Warsaw  Pact,  and  the  political  status  in  the  communist 
Eastern  Europe  Countries,  the  reunification  of  Germany, 
etc.  have  caused  a  somewhat  relaxed  military  posture 
that,  in  spite  of  persistent  small  conflicts  and  areas  of 
latent  crisis,  has  driven  NATO  and  its  Member  Nations 
to  a  decrease  in  their  military  commitment  and  budgets. 

We  hope,  that  these  budget  cuts  will  not  affect  AGARD 
resources  and  therefore  AGARD  activities.  Research  and 
Development  in  the  aerospace  field  has  a  beneficial 
impact  in  many  other  areas  of  science  and  technology. 

The  results  of  these  advances  in  technology  are  only 
realized  in  a  long  range  evaluation.  R  &  D  efforts 
cannot  be  slowed  as  cuts  in  these  areas  can  cause  long 
range  problems. 

The  subject  of  the  Symposium  to  be  opened  today  is 
"Advanced  Aircraft  Interfaces:  The  Machine  Side  of  the 
man-Machine  Interface.  This  is  not  easily  translated  in 
Spanish  but  we  could  say  :  El  Lado  Maquina  en  la 
Tecnologia  de  la  Relacion  Hombre -Maquina  en  los 
Nuevos  Aviones. 

The  extreme  interest  of  this  subject  is  focused  in  the 
growing  complexity  of  the  modem  Air -Land  Battle.  This 
places  a  large  workload  on  the  pilot  and  crew  of  combat 
aircraft.  Nevertheless,  new  technologies,  offer  potential 
solutions  to  most  problems  resulting  from  these  increased 
workloads. 

The  32  papers  to  be  presented  here  appear  to  be  to  be 
extremely  promising  and  attest  to  the  high  professional 
qualities  of  the  lecturers. 

I  am  sure  that  this  meeting  will  result  in  a  most 
profitable  event  for  everyone. 

I  want  to  transmit  my  best  wishes  for  a  successful 
meeting  and  also  a  very  pleasant  slay  in  Madrid. 

Ladies,  Gentlemen,  welcome  to  this  Symposium- 

Thank  you. 
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KEYNOTE  ADDRESS 
by 

Cristobal  Marti'n-Rico 
Managing  Director 

CESELSA,  Simulation  &  Avionics  Branch 
La  Granja  84,  Alcobendas 
28100  Madrid  -Spain- 


Chairman,  ladies  and  gentlemen, 

It  is  a  great  pleasure  and  honour  to  be  given  the 
opportunity  to  address  this  distinguished  attendance 
to  the  AGARD  Symposium  on  Advanced  Aircraft 
Man  Machine  Interfaces,  with  emphasis  on  the 
machine  side  of  the  interface. 

This  is  the  first  Symposium  of  the  Avionics  Panel 
in  Spain,  and  it  happens  in  a  year  that  Spaniards 
have  come  to  think  as  a  Wonder  Year,  since  we 
have  Universal  Expo  in  Sevilla,  we  have  the 
Olympics  and  Madrid  is  the  Cultural  Capital  of 
Europe. 

It  is  indeed  a  honour  because  I  believe  Aircraft 
Man  Machine  Interfacing  is  one  of  the  most 
important  and  creative  challenges,  of  technology 
today.  I  have  to  say  that  not  everybody  in  the 
aerospace  industry  gives  adequate  regard  to  this 
topic,  and  some  think  that  falls  within  the 
superfluous  good-to-have  items  brought  around 
with  the  requirement  for  computers.  I  am  sure  this 
views  are  rapidly  disappearing. 

What  is  one  of  the  fundamental  problems  associated 
with  airborne  man-machine  interfaces? 

The  modem  high  performance  military  aircraft  or 
helicopter  is  the  result  of  very  fast  development, 
instated  by  the  actual  or  estimated  threat  of 
potential  adversaries.  It  is  now  the  basic  element  of 
a  very  sophisticated  weapon  system  in  the  hands  of 
the  pilot,  with  tremendous  firepower,  very  fast  and 
agile. 

The  pilot,  as  a  human  being,  is  a  marvel  of 
physical  adaptation  to  walking  on  two  feet  at  low 
speeds,  on  the  surface  of  the  earth,  mostly  in 


daylight.  Over  centuries  he  has  been  a  hunter,  and 
his  vision,  audition,  smelling,  and  kinestesic  senses 
have  evolved  to  make  him  effective  and  also  to 
allow  his  self  protection,  in  the  natural  hunting 
environment. 

The  skills  to  survive  in  our  highly  competitive 
world,  be  it  at  war  or  in  peace,  are  determined  by 
evolution  of  his  mental  abilities  and  not  his  physical 
or  strictly  sensory  qualities. 

We  can,  and  do,  struggle  to  analyze  and  understand 
how  humans  behave,  perceive,  leam  and  perform 
missions  or  tasks.  We  can  conceive  training 
processes  but  we  cannot  change  natural  limits. 

So  the  fundamental  problem  is  to  adapt  the  aircraft 
system  so  that  it  provides  a  viable  interface  to  the 
real  pilot  under  the  extremely  varying  environments 
in  which  he  has  to  perform,  including  night  or  day, 
adverse  weather  effects,  and  extreme  mission 
requirements  and  threat  intensity,  with  acceptable 
levels  of  safety  and  mission  success. 

This  problem  has  received  in  the  last  15  years 
increasing  attention  and  this  Symposium  and  others 
that  have  taken  place,  and  are  to  be  held  by 
AGARD  in  the  next  future,  show  this. 

It  is,  perhaps,  not  peculiar  to  aircraft,  but  the 
distinctive  characteristic  of  the  fast  timing  give  it  a 
difierent  dimension. 

It  is  moreover  a  reality  that  the  problem  remains 
unsolved  and  some  review  of  the  facts  about 
military  aircraft  non-combat  losses  in  recent  crises 
and  exercises  show  the  urgency  and  the  extent  of 
the  effort  to  be  done,  if  present  day  military 
aircraft  are  to  cope  with  Air  Staff  mission 
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specifications.  (Note  that  modem  commercial 
aircraft  losses  should  add  to  the  concerns). 

When  we  compare  the  incidents  and  accidents 
derived  from  structural,  engine  or  avionics  failures 
with  those  generated  by  human  errors  or  Human 
Factors  failures,  the  picture  is  really  a  grey  one  for 
the  contribution  of  this  technology  to  safety  and 
efficiency. 

I  want  to  recall  a  statement  of  the  Lt.  Cnel 
Simonsen  of  the  RDAF  at  the  AGARD  Lecture  on 
Visual  Effects  of  the  High  Performance  Cockpit: 
"We  have  probably  not  seen  one  single  type  of 
aircraft  without  at  least  one  major  human  factors 
design  deficiency". 

So  let  me  say  as  a  first  remark  that  the  subject  of 
the  conference  during  these  days  is  a  very  acute 
problem  and  a  very  important  one  in  terms  of  life 
risk  and  mission  success. 

*  The  workload  problem. 

But  is  it  really  so  complex  a  problem??  Is  it  not 
just  ergonomics  and  space  allocation  in  the 
cockpit??  Our  answer  is  the  subject  has  indeed 
great  complexity. 

The  military  relevance  of  aircraft  weapon  systems 
make  them  the  target  of  an  impressive 
concentration  of  ground  and  air  threats.  The 
pressure  for  more  and  more  sophisticated  weapon 
system  features  and  capabilities,  reflect  ultimately 
in  pressure  on  the  pilot  performance,  as  operator  of 
the  aircraft  and  the  system. 

This  typically  results  in  tasks  of  the  pilot  that 
combine 


Tactical  Decision  Making  and  Tactical 
Situation  Assessment,  obviously  at  combat 
speeds  and  in  three  dimensions, 
continuous  tactical  planning  and  replanning 
and  action  execution. 


Sensor  management.  Sensor  Selection,  and 
data  assesment  and  weighting. 

Vigilance  and  Search;  Detection, 
Recognition,  Identification  of  expected  or 
unexpected  targets  or  threats  or  events. 


Monitoring  the  aircraft  systems  for  proper 
functioning. 

Control,  either  mediated  or  direct,  of  the 
aircraft,  and  weapons. 

Communications  with  formation  or 
command. 

Depending  on  the  mission  and  mission  phase  this 
workload  is  or  can  be  largely  incompatible  with 
pilot  performance  capabilities,  and  leads  to 
potential  break  down  specially  when  coupled  with 
adverse  sensory  conditions,  physical  stress  and 
spatial  disorientation. 

What  can  be  done  ??  It  is  clear  that  basic  elements 
of  the  solution  are  Cockpit  automation  and  Pilot 
computer  aiding  in  order  to  relieve  some  of  this 
workload,  in  new  aircraft  designs  but  also  in  older 
aircraft,  through  modernization,  if  the  actual 
environment  in  which  they  are  used  was  not 
considered  in  their  design. 

The  underlying  Man-machine  interface  technology 
is  just  emerging  today  with  major  improvement 
underway  at  System.  Subsystem  and  Device 
Levels,  which  will  be  mentioned  later. 

*  The  human  side  of  the  Interface. 

But  let  us  not  forget  that  we  are  talking  interfacing. 
We  need  to  think  for  some  minutes  about  the  other 
side  of  the  interface!. 

All  the  improvements  have  to  be  derived,  in  any 
case,  from  an  understanding  of  human  skilled 
behavior  and  how  it  is  organized  in  pilot 
performance. 

From  a  System  point  of  view  the  human  operator 
has  Sensors  such  as  Visual  and  Auditive,  effectors 
such  as  Voice  or  Hand,  and  a  Information 
Processor  where  mental  behavior  is  provided. 

The  Human  Information  Processor  can  be 
represented  simplistically  as  structured  around  a 
large,  relatively  slow  access,  long  term  memory, 
and  three  processing  elements:  a  Perceptual 
Processor,  a  Motor  Processor,  and  a  cognitive 
Processor,  all  working,  on  a  fast  access  and  small 
size  short  term  memory  that  is  the  working 
memory,  which  is  linked  under  cognitive  processor 
control  to  long  term  memory,  this  working  memory 
being  linked  under  perceptive  processor  control  to 


sensors. 
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The  processors  have  a  characteristic  time  on  the 
order  to  100  msec,  and  working  memory  has  a 
typical  volatility  period  of  about  half  minute. 

The  Sensor  and  Processor  model  allows  us  to 
rationalize  on  the  basic  known  facts  that  human 
operator  activity  is  limited  by  physical  sensor 
configuration  (visual,  auditory,  tactile)  and 
processing  capability,  data  set  size  and  throughput, 
but  within  those  limit  has  multitask  capability. 

Skill  acquisition  and  execution  is  the  basic  tool  of 
man  adaptation  to  the  increasingly  demanding  tasks 
that  he  is  requested  to  perform,  and  allows  fast  and 
efficient  reaction. 

It  is  more  difficult  to  model  in  a  simple  way  the 
role  of  skill  acquisition  and  how  skilled 
performance  is  actually  executed,  but  it  can  be 
visualized  as  a  very  efficient  way  to  retrieve  or 
access  large  pieces  of  information  such  as  data, 
procedures,  or  methods  from  long  term  memory  by 
using  associative  mechanisms  generated  by 
experience  and  training. 

Skilled  Human  operators  are  capable  of  handling 
complex  tasks. 

For  example  in  the  case  of  a  continuous  control 
tasks,  when  the  order  of  the  system  to  be  controlled 
is  high  and  the  characteristic  frequencies  are  too 
high,  the  skilled  operator  generates  a  virtual 
reduced  order  model  that  he  is  able  to  control 
within  the  processing  throughput  limits  and 
generally  achieves  stable  control  with  reduced 
accuracy. 

In  sequential  tasks  when  working  memory 
requirements  are  excessive  he  skilled  operator  tends 
to  break  down  the  whole  task  in  unit  tasks  so  that, 
the  memory  requirements  can  be  fulfilled. 

Unit  task  decomposition  is  at  the  very  root  of  the 
skill  adquision  cycle.  Just  remember  how  people 
used  to  perform  by  hand  arithmetic  divisions  by 
breaking  down  the  total  task  (the  full  number  to  be 
divided)  in  smaller  ones  and  writing  down  the 
intermediate  results  so  that  the  next  step  could  be 
done.  If  you  visualize  this  example  you  realize  the 
conditions  for  unit  task  decomposition  that  are 

.  Working  Memory  Capacity  for  each  step. 


.  Information  Horizons 
Data:  What  you  see  is  what  you  need. 

Method:  What  to  do  each  step. 

Task:  What  boundaries,  input  and  output 

.  Error  Control. 

Skill  acquisition  provides  for  the  evolution  of 
human  mental  behaviour  through  its  skill 
dimension. 

At  its  origin  there  is  the  pure  problem  -solving 
behavior  characteristic  of  new/  improve  situations 
where  man  makes  trial  and  error  steps  in  order  to 
achieve  a  goal  that  he  has  set.  With  experience  and 
training  he  travels  along  this  dimension  gaining 
skill,  that  is,  developing  strategy,  methodology,  and 
ultimately  memory.  Depending  on  the  task  nature, 
environment,  and  timing  and  memory  requirements 
extreme  skill  can  be  developed  with  automaticitv  or 
the  behavior  remains  a  problem-solving  one,  or 
what  is  the  most  usual  case,  the  task  is  broken 
down  into  several  subtasks  falling  each  one  into 
either  category. 

Problem-Solving  behaviour  even  if  slow  is  what 
makes  man’s  contribution  essential. 

Having  said  ail  that  about  the  other  side  of  the  man 
machine  interface,  what  is  it  possible  to  do  on  the 
machine  side  to  reduce  multiple  task  workload? 

I  would  say  actions  can  be  categorized  into  task 
design  "optimization  for  sharing"  and  pure 
automation. 


*  Task  design. 

About  task  design  four  clear  directions  should  be 
pointed. 

First  is  task  resource  demands  reduction  by 
improving  those  task  characteristics  that  are  most 
resource  demanding,  like  for  example: 

Sensory 

Visual  clutter  and  display  resolution 
Readability  of  symbology. 

Auditory  display  clarity. 

Auditory  display  noise. 

Tactile  smoothness  and  feedback. 
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Processing 

Memory  retention  duration  and  size. 
Phonetic  and  semantic  confusability. 
Response  Frequency  and  Complexity. 
Degree  of  choice. 

Motor 

Stimulations-Response  Incompability. 
Error  intolerance  and  accuracy 
requirement. 

Overall 

Controlled  system  order  and  bandwidth. 
Control  gain. 

Tolerable  delay 

A  second  element  is  to  facilitate  task  parallel 
processing  by  using  for  concurrent  tasks,  wherever 
possible,  separate  channels  at  the  sensory, 
processing  and  motor  stages.  Therefore  task  design 
should  take  advantage  of  shareability  of 

Peripheral  modes 

Auditory  vs.  Visual  Perception 

Voice  vs.  Manual  Control 

Encoding 

Verbal  vs.  Spatial  Encoding 
Processing 

.  Perceptual  Cognitive  vs. 

Perceptual  Motor. 

Third  is  task  synergy  and  integration. 

Synergy  can  be  exploited  at  sensory,  and 
control  levels  by  concentrating  visual 
inputs  that  may  have  to  be  perceived 
concurrently  like  for  example  is  done  with 
target  and  own  aircraft  data  in  helmet 
mounted  displays. 

In  this  case  display  information  of  various 
origins  in  a  way  that  leads  to  integrated 
processing  can  be  very  effective  load 
reduction  device. 

Finally  one  of  the  most  effective  task  design 
techniques  is  to  define  and  appropriately  support 
task  decomposition  into  unit  tasks. 


*  Cockpit  automation. 

In  the  cockpit  we  find  initially  three  different  kinds 
of  automation:  Control  Automation  such  as 
autopilots  and  flight  Management  Systems; 
Detection  Automation  including  topics  like  Sensor 
fusion  but  also  alert  and  warning  generation;  and 
Decision  and  Assessment  Automation  that  includes 
decision  aiding,  situation  assessment  and  diagnosis. 

The  problem  of  replacing  human  performance 
versus  assisting,  and  how,  the  human  operator  is  a 
difficult  one. 

Generally  speaking  if  we  look  at  the  continuous 
dimension  of  problem-solving/skilled  behavior  or 
knowledge  based  versus  rule  based  performance  it 
is  clear  that  an  activity  that  human  can  perform 
based  on  pure  skill  can  be  automated.  There  are 
little  number  of  those  at  the  higher  levels  of 
complexity  and  interactivity,  typical  of  the 
unpredictable  combat  environment  but  clearly 
progress  has  been  done,  and  there  is  still  ample 
room  for  additional  automation. 

On  the  other  hand  automated  systems  can  become 
complex  and  therefore  complicated  to  set  up  and 
operate  without  critical  risk  of  error.  So  in  many 
cases  automation  is  to  be  partial,  for  certain 
subtasks  of  a  whole,  and  therefore  requires  abstract 
man  machine  interaction. 

Such  interaction  has  been  implemented  through  the 
ubiquitous  use  of  the  CRT  screens  and 
multifunction  displays,  and  will  probably  extend 
into  "Big  picture”  and  helmet  mounted  displays, 
with  possibly  increasing  use  of  voice-auditory 
commands  and  inputs. 

Whatever  the  physical  interaction  medium,  the  type 
and  format  of  the  symbology,  either  verbal  or 
spatial,  remains  a  key  issue  associated  with 
decluttering,  scheduling,  and  the  difficult  topic  of 
supervisory  control  acceptance  by  the  pilot. 

*  Adaptive  Automation. 

A  word  should  be  said  about  Adaptive  Automation 
and  Interfacing,  dealing  with  system  in  which  the 
degree  of  automation  is  tailored  to  he  varying  needs 
of  the  pilot  or  human  operator. 

Do  we  know  what  the  pilot  thinks  or  needs?. 
Adaptation  to  the  individual  requires  to  identify  the 
cognitive  state  of  the  human  and  devise  at  every 
point  in  time  what  he  needs;  and  supply  that  help. 


and  only  that.  For  instance  analyzing  eye  fixation 
can  give  information  on  what  the  pilot  has 
remarked  and  allow  to  supply  related  information. 

Adaptation  to  the  situation  or  the  external 
circumstance  requires  to  identify  such  situation  and 
provide  the  type  of  automation  and  information 
display  that  is  best  suited  for  it.  This  is  increasingly 
being  used  through  automatic  moding  and  context 
detection. 

System  Instability  and  system  failure  recovery  are 
areas  to  research  and  solve. 

*  Conclusion. 

As  conclusion,  man  with  all  his  physical  and 
functional  limitations,  with  error  making  and 
disorientation  tendencies,  with  attention  failures  and 
sensory  and  psychological  inadaptation  is  the  one 
necessary  piece  of  the  military  aircraft  and  the  key 
to  its  combat  superiority. 

Task  design  and  automation  should  always  be 
directed  towards  aiding  such  limitations  and 
allowing  maximum  use  of  his  capabilities. 

1  have  tried  to  introduce  some  key  topics  of  the 
subject  and  1  hope,  ladies  an  gentlemen,  that  you 
have  an  interesting  symposium  and  a  nice  stay  in 
Madrid. 
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SUMMARY 

The  applications  and  limitations  of  high 
resolution  afforded  by  modern  display 
technologies  are  discussed,  in  relation  to 
the  properties  of  the  human  visual  system; 
and  how  much  'engineering’  may  become 
possible  early  in  the  design  phase  by  the 
use  of  model (s)  of  the  'visual  system  - 
technology  interface’  (VSTI).  Display 
technology  models  provide  good  predictions 
of  the  distributions  of  luminance,  color 
and  contrast  under  specified  driving  and 
environmental  conditions.  Coupled  to 
suitable  vision  models,  estimates  of 
visibility  of  pattern  details  can  be  made. 

In  VSTI  models,  the  beholder  of  the  imagery 
is  regarded  as  a  detector  responding  to 
displayed  patterns  with  'yes' ,  'no'  or  even 
be  allowed  fuzzy  and  false  responses.  Some 
conclusions  are  given  concerning  design  of 
pattern  details  in  imagery,  given  the 
characteristics  of  the  display  and  of  the 
observer . 

LIST  OF  SYMBOLS  AND  ABBREVIATIONS 

A  apparent  area  of  observed  object 
B  brightness 
AB  brightness  Interval 

contrast  ratio:  foreground  luminance 
Lf  over  background  luminance  L 
D  area  of  (eye)  pupil 

L  luminance,  luminous  flux  per  solid 

angle  and  per  surface  area  [nit] 

AL  luminance  difference 
ra  modulation  index 
p  pitch  of  display  elements 
P  active  area  in  V.  of  display  element 
<r  standard  deviation  of  distribution 
t  time,  time  duration 
v  visibility 

x,y  spatial  variables  at  the  display  face 

CRT  cathode  ray  tube 

del  display  element 

EL  electro  luminescence 

JND  Just  noticeable  difference 

LC  liquid  crystal 

LED  light  emitting  diode 

MMI  man  machine  interface 

pel  picture  element  (also  pixel) 

PSF  point  spread  function 

VSTI  visual  system  -  technology  interface 

2-D  variables  extending  In  2  dimensions 

1  INTRODUCTION 

The  combination  of  digital  electronics  and 
new  display  technologies  offer  programmable 
high  resolution,  color  display  devices: 

Head  Down,  Head  Up,  Helmet  Mounted,  Wide 
Field  of  View,  Super  Imposed,  etcetera. 

Power  consumption  is  very  moderate;  weight 
low  compared  to  the  performance;  thermal 
and  electromagnetic  radiation  passable. 
Advanced  graphics  and  image  processing 
software  afford  new  creative  opportunities 
in  imagery  design,  with  strong  appeal  to 
man's  Inborn  cognitive  abilities. 


Fast  perception  of  information  (in  contrast 
to  raw  data)  is  the  issue. 

We  now  see  more  ’natural’  coding  in  the 
man-machine-interface  (MMI):  improved 
matching  of  the  ’problem  data’  formats  to 
the  style  of  human  ’ inner  representation’ 
(geometries  and  contrasts  with  very  good 
cognitive  characteristics,  identifiers 
such  as  generally  valid  geometry  rules, 
color,  positions,  pattern  labels);  and  by 
’fooling  the  eye’  through  adaption  of  the 
spatial  and  metrical  features  of  stimuli 
(like  the  OCR  font  was  designed  for 
machine  vision)  to  the  operative  states 
of  the  ’parameters’  of  the  vision  system, 
induced  by  the  existing  environmental 
conditions. 

Fast  and  reliable  acquisition  of  image  data 
at  low  probabilities  of  confusion  requires 
that  the  brain  be  presented  with  ’adequate’ 
pattern  stimuli.  Important  is  visibility  v. 
here  defined  as  the  probability  that  the 
eye  receives  sufficient  luminous  energies 
to  send  ’adequate’  signals  to  the  brain. 
Factors  involved  are:  luminance  L  and 
integration  time  t;  pupil  diameter  D; 
apparent  size  A  of  the  object  and  local 
spatial  contrast  distribution  C  (x.y)  to 
exceed  thresholds  of  seeing. 

Low  visibility  takes  time. 

How  fine  should  be  the  displayed  ’artwork’? 
Consider  rather  intelligent  machines  able 
to  calculate  the  success  probabilities  of 
a  choice  of  operations,  in  the  order  of 
their  priorities;  and  advising  the  pilot 
accordingly.  It  is  conceivable  that  the 
messages  of  such  a  machine  do  not  need 
very  high  resolution,  in  contrast  to 
imagery  In  the  current  situation  where 
all  the  Intelligence  is  concentrated  in 
the  pilot.  Thus,  expensive  machine 
’Intelligence’  can  probably  be  traded  for 
the  considerable  cost  of  (often  only 
locally  required)  high  resolution  in 
large  surface  area  displays. 

For  the  time  being  the  driving  forces  in 
display  R&D  are  towards  large,  adaptive, 
large  contrast  range,  color  displays  with 
high  resolution. 

Good  visibility  is  required  for  both  the 
"written"  features  of  a  symbol  and  for  the 
"empty"  or  "nonwrltten"  spaces  in  between. 
Easily  confused  image  features  share 
similarity  factors  (both  cognitive  &nd 
psychophysical)  with  the  desired  feature. 

It  depends  on  the  distances  in  the  feature 
space  whether  false  responses,  i.e.  pattern 
confusions,  occur.  Optimal  use  of  cognition 
is  a  matter  of  image  design,  not  of  display 
technology.  Therefore,  our  analysis  is 
restricted  to  high  probability  of  detection 
and,  to  keep  matters  relatively  simple,  to 
the  effect  of  brightness  only. 

Complicated  images  are  formed  by  local 
combinations  of  elementary  patterns  which, 
in  turn,  are  made  up  of  dels. 

Gray  levels  are  used  for  several  reasons: 
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-  to  attract  attention  or,  Inversely,  to 
normalize  brightness  of  differing  colors; 

-  to  fool  the  eye  Into  seeing  smooth  lines, 
sub-del  edge  displacements  and  varying 
line  thickness; 

-  to  normalize  the  perception  of  subtle 
pattern  features. 

But  in  many  cases  the  combined  requirements 
on  contrast  and  luminance  ranges  (gray 
levels,  dimming;  good  visibility  under  all 
circumstances,  little  glare,  no  blinding) 
often  are  difficult  to  satisfy,  unless  the 
Imagery  Is  designed  for  robustness. 

The  Inverse  solution  would  be  to  control 
visibility  by  excluding  changes  in  VSTI 
conditions,  like  in  the  windowless 
cockpit.  Then,  and  even  at  moderate  cost, 
every  refinement  is  possible  in  visual 
transfer  of  Information  (but  also 
necessary!);  and  large  panel  area  and 
high  resolution  and  a  wide  color  gamut. 

■Perceived  image  quality  is  neither  assured 
by  global  technical  specifications  of  the 
display  technology  or  the  specific  product, 
nor  by  general  ergonomic  considerations 
applied  in  the  design  phase  of  the  display. 
In  particular,  the  visibility  of  displayed 
patterns  depends  on  geometries,  on  very 
local  contrast  and  color  differences  and  on 
the  adaptation  state  of  the  eye. 

In  terms  of  detector  models;  sufficient 
changes  in  local  brightness  and  color  to  be 
detected  with  high  probability,  e.g.  >99 
Low  probability  of  confusion  needs  adequate 
visibility  of  the  Individual  features. 
Therefore,  it  is  necessary  to  study  the 
factors  determining  visibility  of  local 
pattern  details.  In  table  I  the  factors  in 
some  way  affecting  visibility  are  marked. 

In  this  paper  the  focus  is  on;  average 
brightness,  line  separation,  stroke  width, 
active  area  and  blur. 

TABLE  I:  VISIBILITY  FACTORS 

-  Del  active  area 

-  Del-to-del  luminance  contrast 

-  Quantisation;  contouring 

-  Brightness  ripple 

-  Whole  Image  vibration 

-  Line  Jumping  due  to  Interlace 

-  Del-to-del  color  contrast 

-  Separation  gap  width 

-  Del/gap  luminance  and  color  contrast 

-  Line  spacing 

-  Stroke  width 

-  Symbol  area 

-  Intra-symbol  area 

-  Symbol  spacing 

-  Reflections 

-  Average  luminance 

-  Dimmability 

-  Adaptation  state  of  the  eye 

-  Dwell  time  of  fixation  (short/gaze) 

-  Flicker  (global  and  local) 

-  Blur 

-  Noise 

-  Shading 

-  Task  dependent  factors 

2  BRIGHTNESS 

2.1  From  luminance  contrast  to  brightness 

Brightness  B,  not  luminance  L,  determines 
the  quality  of  perception.  The  definition 
of  visibility  v  can  be  redefined  as  the 


probability  that  the  accumulated  photon 
fluxes  In  the  pattern  spatial  distribution 
L(x,y),  received  during  a  given  time 
interval  t  (dwell  time  of  1  fixation,  see 
figure  1),  produce  brightness  changes  AB  in 
the  resulting  brightness  distribution 
B(x,y)  sufficiently  strong  to  to  enable 
interpretation  by  the  brain. 


Fig.  1)  Example  of  visual  scanning  of  text: 

fixations  (dwell  time  >  0.25  s)  and 
saccades  in  between  (<  3  rad/s) 

The  number  of  photons  raining  on  the  retina 
depends  on  the  variables  A,L,D,t.  The 
brightness  changes  AB  are  modeled  by  two 
separable  functions:  AB  =  f(A,L,D,t).g(C  ) 
with  0<  f ( A, L, D, t )  <1.  r 

The  target  apparent  area  determines  not 
only  the  number  of  photons  exiting  from  the 
surface,  but  also  the  contrast  loss  due  to 
point  spreading  in  the  optical  part  of  the 
eye  (smearing,  reduces  the  number  of 
photons  at  the  axis  of  viewing).  This  loss 
is  partly  restored  by  neural  action  in  the 
retinal  tissue.  Thus  the  spatial  brightness 
distribution  AB(x,y)  of  the  pattern  results 
from  its  luminance  distribution  L(x,y)  by  a 
mapping  in  early  visual  processing. 

Perceived  sensation  of  luminous  contrast  is 
rated  by  counting  the  number  of  threshold 
brightness  increments  (JNDs,  for  which  by 
definition  AB  =  1 )  which  fit  in  the 
interval  AB  associated  with  the  luminous 
contrast . 

The  JND  is  measured  e.g.  as  a  50  %  response 
probability  to  the  luminance  change.  This 
response  level  is  also  chosen  for  the 
’visibility’  v(AB=  1)  (threshold  of  seeing). 
Blackwell  (1946),  measuring  the  JND  using 
disc  shaped  targets,  found  that  the 
threshold  visual  contrast  AL/L  =  0.0027 
over  wide  ranges  of  disc  diameters  and 
luminance,  in  agreement  with  Weber’s  law: 
AL/L  is  constant.  Systematic  departure  is 
experienced  at  small  target  area,  low 
light  conditions,  short  exposure. 

Weber’s  law  suggests  a  logarithmic  function 
coupling  brightness  to  luminance;  regional 

AB  ~  370. In  C  . f (A,L  ,D,t).  (2.1) 

r  b 

2.2  Visibility 

Visibility  v  is  a  saturating  function:  the 
wide  range  of  the  brightness  interval  scale 
is  non-linearly  mapped  onto  a  visibility 
range  of  0  <  v  <  1 . 

For  long  (many  eye  PSFs  involved)  edges, 
separating  two  uniform  fields,  this  mapping 
can  be  modelled  by  the  error  function: 

U 

erf  u  =  2/Vn  S  exp  (-t2).dt. 
o 
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Remembering  that  the  visibility  at  AB  =  1 
equals  50  the  model  for  the  visibility  v 
associated  with  brightness  interval  AB  is: 

v  =  erf  (0.48  Int|AB| )  (2.2) 

For  AB  =  3  JND,  v  already  attains  0.96. 
Figure  2  combines  the  variables  luminance, 
brightness  and  visibility.  Depicted  are: 

+ 

Cumulative  .. 
brightnes 
Intervals  AB 

B  +  7.  AB  - >' '  ♦ 

b 


Luminance  L  of  Visibility  v 
observed  areas 


Fig.  2)  Mapping  of  luminance  contrast  via 
brightness  intervals  into  probabi 
llty  of  seeing  that  contrast 


-  To  each  increment  AB  =  1  JND  corresponds 
a  ratio  AL/L^Weber1  s  law),  provided  the 
average  luminance  is  sufficient  (photopic 
range,  not  photon  noise-limited). 

-  The  ’zero'  of  the  brightness/visibility 
function  shifts  to  the  brightness  level 
corresponding  with  the  background 
luminance  L  . 

The  parameters  object  area, pupil  size  and 
dwell  time  in  f(A,L  ,D,t)  are  omitted; 
likewise  the  'adaptation  state  of  the  eye’. 

But  for  small  objects  (local  regions  in  the 
image,  e.g.  small  discs,  triangles,  bars) 
the  threshold  ratio  AL/L  is  not  invariant; 
on  top  of  the  effect  of  received  photon 
energies  it  is  also  a  function  of  the 
geometry  because  of  locally  operating 
'lateral  Inhibition’,  a  neural  contrast 
enhancement  process  on  the  receptor 
outputs. 

In  complicated  patterns  (such  as  in  flight 
symbology  and  text)  the  perceived  local 
contrast  thus  depends  on  size/area  of  the 
patterns/features,  visibility  analysis  must 
be  based  on  (2-D)  brightness  distribution 
calculations,  using  an  engineering  model  of 
vision  describing  early  visual  processing 
in  the  eye. 

Therefore  the  statement  by  Calves  &  Brun 
(1976),  that  for  visibility  assurance  at 
least  7  JNDs  are  required  (AB  >  7  threshold 
brightness  increments:  C  >  1.02)  is  not 
sufficiently  precise.  Also  it  is  not  valid 
at  the  very  low  end  of  the  photonic  range 
into  the  scotopic  range  where  every  photon 
counts  (should  be  counted  by  the  receptors 
In  the  retina);  where  nature  Invokes  energy 
preserving  additional  processes. 

The  appearance  of  the  Images  presented  by 
display  devices  also  depends  on  the  chosen 
technology:  consequently  the  visibility  of 
pattern  details  also.  We  arrive  at  the 


computation  structure  of  figure  3.  The 
remainder  of  the  paper  focuses  on  the  third 
part:  the  vision  model. 

2.3  A  vision  model 


The  eye  is  modeled  as  an  early  visual 
processor,  responding  to  local  contrasts  by 
corresponding  numbers  of  equal  brightness 
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Fig.  3)  Modeling  the  visibility  of  images 

displayed  in  a  specified  technology 


increments  allowed  to  also  produce  false 
responses.  In  figure  4,  one  possible 
engineering  model  is  depicted  (Bakker  and 
Bosman,  1988). 

It  is  composed  of: 

-an  optical  part,  representing  the  eye 
optical  system,  accounting  for  aberration 
of  the  lens,  pupil  diffraction  and  scatter 
in  the  mess  of  nerve  bundles  and  blood 
vessels  just  in  front  of  the  retina;  the 
result  being  smearing  of  the  input  image, 
-a  detector  part,  representing  the  eye 
neural  system  which  resolves  the  smeared 
illuminance  distribution  at  the  retina 
Into  'sharp'  brightness  signals  sent  to 
the  brain,  taking  account  of  photon  noise 
and  neural  noise. 


Fig.  4)  The  blockdiagram  of  the  vision 
model 

The  detector  part  consists  of  two  branches, 
with  different  smearing  properties  (center 
with  low  smearing,  surround  with  strong 
smearing);  both  perform  a  log  operation. 

All  'Point  Spread  Functions’  (PSFs) 
involved  are  not  Invariant  but  depend  on 
internal  states  and  environmental  luminous 
conditions. 

Note  that  the  log  operation  followed  by 
subtraction  of  the  two  2-D  Images  yields  a 
brightness  Interval  2-D  distribetlon 
proportional  to  the  luminance  contrast 
ratio  distribution;  while  simultaneously 
performing  enhancement  because  the  (wide) 
skirt  of  the  optical  PSF  Is  removed  by  the 
lower  branch  PSF  (known  In  Image  processing 
by  the  name  'unsharp  masking’,  in  vision 
literature  as  'lateral  inhibition'. 
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Photons  available  In  a  point  source  are 
distributed  by  a  PSF  to  fill  Its  entire 
volume  (Westhelmer  G.  &  Campbell  F.W. 

1962),  thereby  lowering  peak  response 
considerably.  For  the  eye  optical  PSFop  its 
peak  magnitude  is  attenuated  70  times  with 
respect  to  absence  of  smearing. 

A  large  area  stimulus  is  composed  of  many 
points,  their  overlapping  responses  add 
their  photon  catches  to  produce  a  uniform 
response  close  to  unity. 

The  responses  of  the  eye  optical  system 
(transfer  function  from  object  to  retina) 
to  several  pattern  primitives  are  depicted 
in  figure  5. 


Fig.  5)  responses  at  the  retina  (receptor 
level)  of  the  eye  optical  system 

a)  a  point  source  (PSFop); 

b)  a  linepiece,  very  thin,  with  length  5 
times  the  width  of  PSFop; 

c)  a  long  line;  and 

d)  square  stimulus  of  5x5  widths  of  PSFop. 
(Note  the  differences  in  peak  magnitude) 

The  spatial  response  of  the  total  model  to 
a  point  stimulus  (eye  PSF)  (Blommaert  and 
Roufs,  1981)  is  depicted  in  figure  6c), 
along  with  the  PSFs  of  the  foreground  path 
(6a)  and  the  background  path  (6b)  (neural 
pooling  PSFs  located  in  the  detector  part 
of  the  model ) .  This  response  is  taken  to  be 
valid  in  the  center  of  the  foveal  region 
(foveola) . 


Fig.  6):  Point  Spread  Functions  in  the 
vision  model,  see  text 

A  small  target,  like  an  alphanumeric 
character  of  6  mm  high  seen  at  0.7  m,  has  a 
viewing  angle  of  30  minutes,  well  within 
the  size  of  the  foveola  where  the  retinal 
resolution  remains  about  invariant.  The 
operation  of  the  model  (optical  smearing, 
lateral  Inhibition  and  mapping  from 
luminance  to  brightness)  is  demonstrated 
in  figure  7,  showing  the  letter  ’A'  as  it 
appears  on  an  electro-luminescnce  display 
and  the  perception  of  brightness. 


Fig.  7)  Mapping  luminance  into  brightness 
by  the  vision  model. 


Towards  the  periphery  the  PSF  aequivalent 
cross  section  increases  (Koenderink  8.  van 
Doom,  1978).  In  figure  8  the  gradual  loss 
of  sharpness  (and  contrast)  (one  fixation) 
in  peripheral  view  is  demonstrated.  By 
taking  several  fixations  across  the  image 
the  brain  builds  up  a  total  sharp  picture. 
Moreover,  memory  from  previous  (almost 
identical)  views  are  thought  to  reduce  the 
required  number  of  fixations. 


Fig.  8)  Loss  of  sharpness  away  from  the 
point  of  regard. 


2.4  Quantitative  examples 

For  non  noise-limited  conditions  (luminance 
>  10  nit)  the  combined  effect  of  variations 
In  luminance,  fixation  time  and  pupil 
diameter  is  small.  The  response  in  number 
of  JNDs  can  be  approximated  by 

AB  »  370. In  C  ,f(A),  (2.3) 

r 

with  f (A)  a  function  of  eg.  the  area  and 
shape  of  the  stimulus.  One  obtains  AB  «  7 
JND  («  100  V,  reliable  detection)  for  large 
area  objects  (f(A)  «  1)  at  a  C  of  1.02. 

BUT,  for  small  stimuli  like  a  short  bar, 
i.e.  a  linear  array  of  5  display  elements 
(figure  5b)  at  the  (high)  resolution  of  12 
dels/mm,  the  eye  PSFop  attenuates  the  input 
contrast  ratio  of  1.02  to  1.0026  at  the 
retina:  its  brightness  interval  AB  equal  to 
1  JND!  Fine  detail  is  barely  visible.  Such 
an  image  looks  ghostly.  In  figure  9  the 
visibility  is  plotted  of  the  5x1  bar  as 
function  of  display  element  size. 

It  is  clear  that,  at  higher  resolutions  or 
in  finer  artwork,  the  display  must  be 
driven  harder  (more  contrast  required). 


To  see  also  points  (i.e.  very  small  dels) 
reliably,  the  contrast  ratio  must  be 
C  =  1  +  70  x  0.02  =  1.4  ! 

In  gray  scale  images  these  considerations 
are  valid  at  all  (local)  background  levels; 
thus  without  ’fooling  the  eye’,  gray  scales 
should  have  contrast  ratio  intervals  of 
minimally  1.4,  in  agreement  with  earlier 
evidence  (Carel  1965). 
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Fig.  9)  Visibility  versus  size  of  the 

display  eleaent  in  pm  for  a  bar 
5  dels  long  and  1  del  wide. 
Viewing  distance:  0.7  m; 
Contrast  ratio:  1.02; 
background  luminance:  100  cd.mm 


calculated  as  shown  in  figure  11.  which 
depicts  the  required  brightness  in  JND  as 
function  of  gap  width.  Consequently,  from 
the  visibility  angle,  character  height  can 
be  quite  small. 

jj  required  AB  (JNDs) 


" - 1 - 1 - 1 - 1 - T 

0  50  100  150  200 

line  separation  [pm]  - » 


100- 

80- 

60- 

40- 

20. 


This  leads  to  excessively  large  luminance 
ranges  (>  250,  in  addition  to  the  required 
dimmabllity  range).  Again  compensation  can 
be  found  either  in  driving  the  display 
elements  adaptively,  or  in  special  designs 
of  ’small’  feature  areas  while  retaining 
full  resolution  to  preserve  image  quality. 


Fig.  11)  Threshold  brightness  of  gaps 
0.7  m  viewing  distance 

3  LOCAL  BRIGHTNESS  MODULATION;  EFFECT 
OF  ACTIVE  AREA  OF  THE  DISPLAY  ELEMENT 

3.1  Modulation  of  pattern  features 


Decreasing  the  viewing  distance  increases 
angular  size,  which  diminishes  optical 
attenuation,  thereby  increasing  contrast  at 
the  retina.  Moreover,  at  higher  luminances 
more  photons  partake  in  the  stimulation 
process  so  that  the  neural  PSF  control  can 
decrease  the  aequlvalent  cross  section 
resulting  in  a  crisper  picture. 

The  desire  to  augment  the  visibility  of 
very  small  features  relates  to  the  reason 
why  people  tend  to  bend  forward  and/or  to 
operate  emissive  displays  at  higher 
luminances. 

In  radiant  stimuli  with  strokes  made  up  of 
several  dels,  visibility  Is  much  higher  due 
to  2-D  merging  of  PSF  responses.  Assuming 
ample  contrast  (AB  »  25  JND)  The  effect  of 
stroke  width  is  depicted  in  figure  10, 
consistent  with  (Reger  &  Snyder  &  Farley, 
1989)  for  9x12  matrix  blocks  or  better. 

At  reduced  angular  symbol  height,  the 
smallest  intra-symbol  spaces  (in  ’a’  and 
’s’)  with  area  <  2'  experience  loss  of 
visibility.  The  same  happens  when  stroke 
width  Increases  at  constant  character 
height,  implying  that  the  curve  of  figure  8 
mirrors  at  thicker  stroke  widths. 

At  positive  contrast  effects  of  PSF  merging 
is  detrimental  for  very  thin  darker  strokes 
and  for  intra-symbol  spaces,  contrast  of 
the  whole  symbol  suffers,  not  only  certain 
features. 


At  lower  resolutions  the  Individual  display 
elements  remain  visible,  causing  modulation 
of  the  pattern  features  shown  in  figure  12. 
The  subtraction  operation  in  the  model  of 
figure  4  is  responsible  for  ’lateral 
inhibition’  producing  ’Mach  bands'  (the 
trough  around  the  peak  in  figure  6c).  In 
figures  12  &  13  its  effect  on  a  complicated 
shape  are  depicted. 


Fig.  12)  Response  of  low  resolution  display 
with  separation  gaps  between  the 
display  elements.  Delpitch  »  500  pm. 


Consequently  (see  also  figure  7): 

A)  ’small’  square  display  elements  seem 
rounded  instead  of  square;  separation 
gaps  widen  at  the  intersections 
increasing  their  visibility. 

B)  with  uniformly  driven  display  elements, 
the  ones  with  few  neighbours  (e.g.  the 
upper  del  of  the  letter  ’A’)  are  seen  at 
increased  brightness,  thus  seem  larger 
than  the  ones  surrounded  by  many 
neighbours; 

C)  the  inhibitory  lateral  action  produces 
also  local  contrast  augmentation  around 
the  features  of  the  pattern. 


Fig.  10)  Visibility  v  versus  stroke  width 
sw  in  pm.  Viewing  distance  0.7  m 

Some  Avionics  displays  use  light  emitting 
diodes,  providing  very  thin  lines  at  high 
brightness.  That  the  visibility  of  very 
small  gaps  in  LED  arrays  is  high,  is 


To  calculate  the  modulation  index,  it  is 
necessary  to  use  the  2-D  vision  model.  By 
the  same  means  it  is  possible  to  design 
reduction  of  image-dependent  unwanted 
modulation,  using  gray  level  blurring 
techniques  discussed  in  section  4  below. 


Fig.  13)  a)  the  3-D  luminance  distribution 
of  a  symbol  A  at  the  face  of  a  EX 
display  with  separation  gaps  of 
1/3  of  the  pitch; 
b)  the  corresponding  brightness 
distribution  of  the  same  symbol, 
obtained  by  the  model,  when  the 
symbol  height  Is  4.5  mm  viewed  at 
a  distance  of  0.7  m. 


A  reasonable  model  to  estimate  modulation 
contrast  In  the  brightness  pattern  of 
adjacent  dels  of  a  line  In  a  main  direction 
of  the  display  Is  the  expression: 

(  no 2  P(p+<r)2+  iktZ  ) 


exp 


(1-P)  (p+<r)  (p+c)  +  no- 


(5.1) 


the  modulation  Index  m=  (B  -B  )/(B  +B  ); 

p  •  p  « 

B  :  peak  brightness,  B  :  saddle  brightness 

p  ■ 

o-  the  s.d.  of  the  eye  PSF. 

A  plot  of  m  as  function  of  p/<r  with 
parameter  P  Is  depicted  in  figure  14. 


Fig. 14)  Effect  of  pitch,  active  area  and 
eye  PSF  on  brightness  modulation 

3.2  The  effective  area  of  a  display  element 

The  ANSI  standard  for  Human  Factors 
Engineering  of  Visual  Display  Terminal 
Workstations  states  that  the  minimum 
permissable  active  area  or  fill  factor  In 
flat  panel  displays  be  30  preferred  is 
75  %  .  From  the  foregoing  It  is  obvious 
that  resulting  brightness  modulation  varies 
with  del  size  and  pitch.  The  recommendation 
thus  Is  not  sufficient  In  itself;  some 
function  of  both  the  number  of  elements  in 
the  character  matrix  and  the  active  area 
per  del  should  be  considered. 

The  resulting  brightness  modulation  then 
depends  on  the  adaptation  state  of  the  eye. 
One  may  specify  a  fixed  'standard'  eye  PSF 
or  use  the  brightness  model  to  calculate 
the  resulting  modulation.  Based  on  the 
assumptions  of  the  model,  under  good 
viewing  conditions  the  modulation  seems 
secondary  as  it  hardly  affects  visibility; 
only  deep  modulations  are  thought  to  have 


effect  but  this  should  be  verified. 

Active  del  area  convolves  with  the  eye  PSF, 
producing  a  modified  PSF  with  effective 
area  about  equal  to  the  sum  of  both.  E.g. 
like  in  figure  12,  with  del  pitch  of  2'  and 
eye  PSF  with  effective  area  of  1.5'  solid 
angle,  the  45  ’/.  active  del  area  results  in 
a  PSF  with  effective  area  of  3.4'  solid 
angle.  The  modulation  contrast  is  clearly 
visible  in  the  brightness  pattern  of  dels 
in  a  line  in  a  main  direction  of  the 
display.  With  75  %  active  del  area  the 
resulting  effective  area  becomes  4.5', 
larger  than  the  pitch  squared  (overlap). 

For  that  reason  modulation  contrast  Is  only 
visible  In  oblique  lines. 

These  considerations  determine  the  required 
resolution  for  uniformity  of  brightness. 


Fig.  15)  Iso-brlghtness  contours  at 

a)  1  JND  -outer/dashed, 

b)  3  JND  -Inner  dashed  and 

c)  7  JND  -solid  line. 

Geometry  of  figure  12, 

C  :  1.4;  L  :  100  nit. 
r  b 


Fig.  16)  Display  iso-brightness  contours: 

a)  electroluminescence, 

b)  twisted  nematic  LC,  backlighted 

The  effects  can  be  studied  by  plotting 
iso-brightness  contours  as  shown  in  figure 
15.  The  contour  at  3  JND  of  15b)  determines 
the  perceived  contour  of  the  symbol.  Note 
modulation  due  to  insufficient  merging  of 
PSFs. 

Annoying?  Who  knows. 

Figure  16  shows  the  iso-brightness  contour 
families  of  symbols  and  background  (7  JND) 
on  two  commercial  displays. 

4  EFFECT  OF  BLUR 

Blurring  (spatial  low  pass  filtering) 
occurs  through  smearing  of  the  designed 
image  with  a  point  spread  function  situated 
either  inside  or  outside  the  eye.  Examples 
of  ’inside’  blurring  are  given  in  figures  6 
and  8.  It  also  occurs  (much  stronger)  in 
neural  pooling  of  receptor  outputs  at  low 
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light  levels  when  necessary  to  lift  the 
total  photon  catch  to  exceed  the  neural 
noise  threshold.  'Outside'  optical  blurring 
occurs  unwillingly  at  the  screen  of  a  CRT 
or  in  display  cover  layers  and  antiglare 
filters,  or  on  purpose  by  electronic  and/or 
algorithmic  means. 

Blur  causes  additional  gray  levels. 

Unwanted  blur  is  most  bothersome  in  sharp 
turns  of  lines;  and  it  narrows  intra-symbol 
spacing,  where  it  should  be  avoided. 

Hamer ly  &  Dvorak  (1981)  pointed  out  that 
edge  gradient  softening  by  0.4'  in  normal 
print  causes  the  experience  of  bold  print. 

In  low  to  medium  resolution  quantised 
displays  blur  can  actually  improve  the 
perception.  In  figure  17  two  approaches  are 
shown:  a)  by  2-D  linear  interpolation 
(Bosnian  &  Umbach,  1982),  b)  by  adding 
extra  graylevels  in  the  direction  of  the 
line  (image  dependent  1-D  interpolation, 
Negroponte,  1980). 


a)  b) 


Fig.  17)  Removal  of  ’ Jaggies’  by 

a)  2-D  linear  interpolation 

b)  1-D  interp.  along  the  line. 

suggesting  a  resolution  of  0.5'.  But  the 
circular  symmetric  eye  PSF  with  effective 
area  of  1.5'  solid  angle  attenuates  fine 
2-D  detail.  The  cut-off  in  resolution 
(single  fixation)  can  be  expected  at  about 
2'  per  del  (2.2  dels/mm  at  0.78  m  viewing 
distance).  At  higher  resolutions  wider  2-D 
excursions  and/or  higher  contrasts  are 
necessary  to  be  perceived  in  the  same  time 
slot.  Correlations  between  gray  levels  of 
display  elements  In  patterns  are  image 
dependent  and  must  be  Individually  defined 
in  each  symbol  matrix. 

Such  intended  ‘blur’  operations,  being 
nonlinear,  can  be  determined  with  the  aid 
of  the  calculated  spatial  brightness 
distribution,  or  determined  in  interactive 
experiments. 

5  CONCLUSION 

Visibility  limitations  occur  in  early 
visual  processing  and  are  very  local.  The 
usual  display  specifications  Involving 
several  degrees  of  viewing  (!)  describe 
only  global  characteristics;  they  cannot 
predict  perceived  symbol  display  quality. 
Such  parameters  either  must  be  measured  by 
ergonomic  experiments  or  can,  with  limited 
accuracy,  be  predicted  by  an  engineering 
model  of  vision  as  argued  in  this  paper. 
Judged  by  the  results  of  our  research,  the 
parameters  currently  used  to  define  display 
standards  (technology  And  Imagery)  are  less 
suitable  to  enforce  display  quality  in  the 
sense  of  good  perception. 
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1.  SUMMARY 

This  paper  identifies  and  discusses  28  problem  areas  where 
human  factors  engineers  lack  the  information  needed  for 
development  of  crewstations  for  advanced  military  fighter 
and  attack  aircraft.  Emphasis  of  this  paper  is  on  naval  air 
missions  projected  during  the  early  21st  century  against  land 
and  sea -surface  targets.  The  28  problem  areas  are  based  on 
the  functions  that  the  crews  must  carry  out  for  successful 
mission  accomplishment.  Human  capabilities  and  limitations 
documented  in  the  human  factors  literature  tha  relate  to 
these  aircrew  functions  have  been  used  to  define  the 
problem  areas.  The  goal  is  to  ensure  that  aircrew 
performance  will  be  satisfactory  for  anticipated  air  missions. 

The  28  problem  areas  are  grouped  into  nine  human  factors 
problem  categories.  For  each  category,  the  human  factors 
knowledge  and  man-machine  interface  engineering 
capabilities  that  should  be  extended  during  this  decade  are 
noted.  The  human  factors  problem  categories  are  (1) 
physical  and  physiological  stress,  (2)  vigilance  and  aircrew 
alerting,  (3)  individual  differences,  (4)  information 
integration,  (5)  visual  displays  for  various  missions,  (6) 
mission  management,  (7)  decision  support,  (8)  automation, 
and  (9)  system  design  and  evaluation. 

2.  BACKGROUND 

2.1  Air  Warfare  in  the  21st  Century 

Constrained  and  limited-objective  warfare  has  received 
renewed  interest  in  the  past  few  years,  as  the  dangers  of 
global  war  have  diminished.  Both  friendly  and  potentially 
hostile  forces  now  own  numerous  kinds  of  weapons  systems 
that  are  nearly  identical,  making  positive  identification 
critical  prior  to  air  attack.  Greater  importance  is  being  given 
to  precision  targeting  and  weapon  placement  that  will 
minimize  civilian  injuries  and  collateral  damage  when 
responding  to  aggressor  and  terrorist  threats  and  attacks. 

These  new  requirements  place  great  burdens  on  the  military 
aircrews  who  must  make  the  final  decisions  about  weapon 
release.  Improved  crewstations  and  advanced  aircrew- 
aircraft  interfaces  of  various  types  have  been  suggested  as 
solutions  to  the  workload  and  decision-making  problems  of 
military  aviators,  to  minimize  errors  and  to  improve  reaction 
time. 

Proposed  advancements  in  crewstation  display  techniques 
include  helmet-mounted  displays,  three-dimensional  voice 
and  other  auditory  advisories,  large  fiat  panel  systems, 
virtual  displays,  and  data  fusion  for  integrated  display 
formats.  System  control  improvements  also  are  proposed: 


advanced  multifunction  switches,  hands-on-throttle-and- 
stick  controls,  voice  commands,  etc.  Intelligent  decision  aids 
and  expert  systems  have  been  offered  as  answers,  along  with 
automation  of  whatever  can  be  automated. 

2.2  Solutions  Versus  Problems 
It  is  important  to  note  that  the  crews  of  advanced  aircraft 
rarely  will  interact  directly  with  the  aircraft  and  its  weapons 
systems.  Instead,  they  will  interact  with  computers  that  in 
turn  control  the  aircraft  systems.  That  is,  the  crews  will  serve 
as  supervisory  controllers  of  the  systems  (Ref.  1).  Therein 
lies  the  promise  of  improving  crew  performance  for  the 
missions  of  the  21st  century,  and  also  the  potential  for 
disaster  if  the  human-computer  interfaces  are  poorly 
designed. 

So  far  most  of  the  emphasis  has  been  on  seeking  isolated, 
piecemeal  solutions  for  the  advanced  aircraft’s  crews, 
without  first  taking  a  systematic  overall  look  at  what  the 
problems  really  are.  What  human  limitations  really  matter 
and  actually  will  result  in  failed  missions?  What  technologies 
are  we  proposing  as  solutions  simply  because  they  are 
feasible — whether  or  not  they  address  :he  most  important 
problems?  An  overview  is  needed,  to  put  the  various 
proposed  solutions  into  perspective,  to  consider  how 
proposed  advances  might  fit  together  or  interact,  and  to  aid 
in  setting  priorities  for  the  technologies  that  should  be 
pursued.  Important  human  factors  issues  and  problems 
should  be  identified,  before  crewstation  designs  are 
solidified. 

3.  PURPOSE  AND  APPROACH 
The  purpose  of  this  study  has  been  to  propose  a  set  of 
human  factors  engineering  issues  specifically  related  to 
advanced  fighter  aircraft  that  should  be  addressed  during 
this  decade.  Based  on  the  kinds  of  naval  air  missions  that 
can  be  anticipated  early  in  the  21st  century  and  on  standard 
advanced  aircraft  systems  and  components  with  which 
aircrews  interact,  the  top-level  functions  that  the  crew  must 
carry  out  have  been  identified.  The  crew’s  basic  information 
and  control  requirements  related  to  these  functions  also 
were  identified. 

Previously  documented  human  factors  books,  reports  and 
articles  on  human  capabilities  and  limitations  were 
reviewed.  Those  capabilities  and  limitations  related  to  the 
aircrew’s  functions  and  information  and  control 
requirements  were  noted.  Based  on  the  resulting  lists, 
potential  problem  areas  that  might  affect  mission 
accomplishment  have  been  identified. 
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4.  MULTI-ROLE  ADVANCED  FIGHTER  AIRCRAFT 
Replacements  for  many  of  today’s  attack  and  fighter  aircraft 
will  be  necessary  in  the  next  few  yiars,  as  current  systems 
become  unable  to  meet  new  requirements.  Next-generation 
tactical  aircraft  often  are  referred  to  as  advanced  fighters, 
emphasizing  the  difficult  air-to-air  combat  missions  they  are 
expected  to  cany  out.  But  reduced  expenditures  for  military 
systems  require  that  most  new  military  aircraft  be  equipped 
to  serve  in  several  air  warfare  roles.  Air  superiority, 
surveillance,  and  attack  roles  all  must  be  considered  for  new 
naval  tactical  aircraft  (Fig.  1). 

A  naval  advanced  aircraft’s  air  superiority  fighter  function 
requires  that  the  aircrew  be  able  to  engage  one  or  more 
airborne  targets  over  land  or  sea,  obtain  information  about 
the  targets  necessary  for  fire  control,  and  direct  one  or  more 
missiles  <o  intercept  the  targets.  Targets  will  include 
adversary  bombers  and  fighters  that  can  be  engaged  either 
(1)  beyond  visual  range  with  long-range  or  medium-range 
radar-guided  weapons  or  (2)  within  visual  range  using  heat¬ 
seeking  missiles  or  an  aircraft  gun.  For  both  situations,  it  is 
important  that  the  aircrew  have  sensors  and  displayed 
sensor  information  designed  to  meet  human  perceptual  and 
cognitive  needs  so  the  crew  will  detect  the  adversary  first 
and  launch  weapons  first. 

Beyond-visual-range  fighter  engagements  usually  are  part  of 
outer  air  battle  missions.  These  engagements  require 
avionics  that  can  assist  the  aircrew  with  weapon  setup, 
targeting,  and  fire  control  tasks.  Although  timeliness  is  not 
as  critical  as  with  some  other  advanced  fighter  functions, 
decision  making  is  especially  difficult  under  beyond-visual- 
range  circumstances,  since  information  usually  is  uncertain. 

Within- visual-range  fighter  engagements  are  expected  to 
predominate  in  most  low  intensity  conflict  and  constrained 
and  limited-objective  warfare  scenarios,  especially  during 
strike  escort  missions.  It  should  be  noted  that  short-range 
engagements  impose  extremely  stringent  requirements  on 
the  aircrew.  Careful  system  design  will  be  required  to 


minimize  aircrew  response  times  and  the  chances  of 
operator  errors  during  the  extreme  time-related  stress  of 
such  engagements. 

In  its  surveillance  role,  the  advanced  fighter  aircrew  must  be 
able  to  search  for,  detect,  assess,  and  use  a  datalink  to 
report  on  an  attack  consisting  of  one  or  more  air  threats 
(potential  targets),  during  the  outer  air  battle.  Again,  this 
function  may  be  performed  under  either  beyond-visual- 
range  or  within-visual-range  conditions,  depending  on  the 
scenario,  rules  of  engagement,  and  nature  of  the  threats.  In 
the  surveillance  role,  the  aircrew  may  be  required  to  spend 
many  hours  in  the  air  in  vigilance  tasks:  searching  for  targets 
and  evaluating  the  combat  situation.  Aircrew  fatigue  will  be 
a  major  problem  while  carrying  out  this  advanced  fighter 
function. 

An  attack  role  also  must  be  anticipated  for  the  naval 
advanced  fighter  aircraft  and  its  crew,  which  must  be 
prepared  to  carry  out  land  or  sea  strike  missions  when 
necessaiy.  The  U.S.  Navy’s  F/A-18  Hornet  aircraft  was 
designed  for  this  dual  role  from  its  inception.  Several  U.S. 
Navy  and  Air  Force  aircraft,  originally  designed  simply  as 
fighters,  in  recent  years  have  been  reconfigured  (or 
reconfiguration  plans  have  been  proposed)  to  handle 
attacks  on  surface  targets  as  well.  These  include  the  F-14,  F- 
15,  and  F-16  aircraft.  Mission  timing  and  route  keeping,  task 
management,  and  decision  making  are  especially  important 
for  the  attack  role. 

5.  ADVANCED  AIRCRAFT  TECHNOLOGIES 
The  term  advanced  fighter  technologies  as  used  here  relates 
to  those  aircraft  systems  and  capabilities  needed  so  that  the 
aircrew  can  carry  out  the  advanced  fighter  missions 
satisfactorily.  At  least  eight  such  technologies  have 
significant  human  factors  engineering  implications,  as 
identified  in  Fig.  2  and  discussed  in  the  following  sections. 
That  is,  whether  aircrews  can  reliably  cany  out  the  various 
functions  required  for  each  system  will  be  affected  by 
certain  basic  human  capabilities  and  limitations  that  have 
been  identified  by  various  researchers. 


Role: 


Tactical 

Operations: 


Range: 


Figure  1.  Roles,  Tactical  Operations,  and  Areas  of  Operation  for  a  Naval  Advanced  Fighter. 


Figure  2.  Advanced  Fighter  Technologies  with  Significant  Human  Factors  Implications. 


What  are  the  functions  that  the  crew  of  advanced  naval 
aircraft  must  cany  out?  A  detailed  listing  of  the  specific 
functions  related  to  each  of  the  eight  kinds  of  systems 
discussed  below  is  beyond  the  scope  of  this  paper.  However, 
a  simple  top-level  description  of  crew  functions  is  useful  for 
understanding  the  kinds  of  human  limitations  and 
capabilities  that  are  important  for  air  missions.  Fig.  3 
provides  an  overview  of  those  functions  and  of  the  crew's 
related  information  and  control  requirements. 

The  human  capabilities  and  limitations  that  are  listed  for 
each  advanced  fighter  technology  are  of  course  not  unique 
to  that  specific  technology  area.  Many  of  the  factors  actually 
will  influence  human  performance  with  several  of  the 
advanced  fighter  systems.  However,  to  avoid  redundancy, 
each  factor  is  listed  only  once,  under  the  system  that  may  be 
the  most  affected  by  that  capability  or  limitation. 

5.1  Avionics  and  Fire  Control  System  Architecture 
For  this  study,  avionics  and  fire  control  system  architecture  is 
defined  as  the  top  level  hardware  and  software  design  that 
combines  the  all  the  subsystems  and  components  that  make 
up  the  advanced  fighter  aircraft’s  avionics  and  fire  control 
systems.  That  is,  the  architecture  consists  of  the 
interconnections,  relationships,  and  organization  of  the 
subsystems  that  together  form  an  advanced  fighter  aircraft. 
Avionics  and  fire  control  system  architectures  should 
provide  top-level  integration  of  all  crewstation  systems,  and 
also  must  be  compatible  with  the  aircraft,  missile  systems, 
and  aircrew  performance  capabilities. 

Various  studies  have  noted  human  capabilities  and 
limitations  that  may  be  affected  by  avionics  and  fire  control 
system  architectures.  These  capabilities  and  limitations 
include  the  following 


1.  Mental  models.  Humans  learn  and  operate  systems 
significantly  more  efficiently  and  effectively  if  they  have 
useful  mental  models  or  ‘‘pictures”  of  how  the  system 
operates  and  about  the  effects  their  own  control  actions 
have  (Ref.  2).  Thus  the  overall  system  architecture  should 
be  describable  in  terms  easily  comprehended  by  naval  pilots 
and  other  crew  members  (Ref.  3). 

2.  Memory  limitations  and  “situational  awareness. " 
Humans  generally  can  retain  only  five  to  nine  items  or 
“chunks”  of  data  in  immediate,  short-term  memory  at  one 
time  (Ref.  4).  When  busy  or  under  stress,  humans  perform 
mud]  worse  than  this;  they  have  difficulty  operating  on 
more  than  a  single  idea  at  a  time.  In  essence,  they  exhibit  a 
form  of  mental  ‘tunnel  vision.”  Provision  should  be 
included  to  ensure  that  the  aircrews  know  what  mission 
tasks  have  been  completed,  and  will  remember  and  perform 
all  remaining  required  tasks,  even  under  the  most  difficult 
circumstances.  This  includes  resuming  system  control 
following  recovery  from  g-induced  loss  of  consciousness 
(GLOC). 

3.  Decision-making.  In  the  beyond-visual-range  fighter 
role,  the  aircrew  must  evaluate  uncertain  information 
provided  by  the  data  fusion  system,  then  determine  whether 
to  engage  long-range  targets.  Decision-making  assistance 
will  be  needed  as  part  of  system  architecture  to  optimize  the 
crew’s  performance  of  this  task  (Ref.  5). 

4.  Time  stress.  In  the  within-visual-range  fighter  role,  the 
aircrew  must  process  information,  make  correct  decisions, 
and  complete  appropriate  responses  with  extreme  rapidity. 
Pilots  restrict  their  “sampling”  of  information  to  what  they 
perceive  as  “most  important" — and  are  not  always  right 
(Ref.  2).  System  architecture  should  include  intuitive 
information  display  and  adaptive  automation  (Ref.  6)  of 
some  control  functions  for  this  fighter  role. 
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1.  Alrcraw  (unction:  Premission  and  midmission  planning. 

Required  Information:  Mission  plans  and  status. 

Required  controls:  Mission  planning  and  plan 
modification  controls. 

2.  Aircrew  function:  Understand  system  components  and 
component  integration. 

Required  Information:  Online  documentation  about 
aircraft  systems,  mission,  and  tactics. 

Required  oontroie:  Controls  to  request  system  architecture 
and  status  information. 

3.  Aircrew  function:  Understand  integrated  information  and 
integration  systems. 

Required  Information:  Integrated  data  from  ail  compatible 
sources. 

Required  oontroie:  Information  output  display  optimization 
controls. 

4.  Aircrew  function:  Top-level  system  control. 

Required  Information:  (a)  Status  of  supervisory  control  of 
all  systems,  (b)  Top-level  status  of  all  systems. 

Required  controie:  Top-level  moding  controls. 

5.  Aircrew  function:  Monitor  and  interpret  aircraft  status 
Information  and  make  decisions. 

Required  information:  Aircraft  attitude,  altitude,  speed, 
energy  and  power,  force  due  to  gravity,  location, 
direction  of  flight  fuel  status. 

Required  controie:  Information  output  optimization 
controls. 

6.  Aircrew  function:  Control  the  aircraft. 

Required  Information:  Current  and  projected  status  of 
aircraft  systems. 

Required  controie:  Controls  for  attitude,  altitude,  airspeed, 
direction  of  flight 

7.  Aircrew  function:  Monitor  and  Interpret  onboard  system 
status  and  mission  status  information  and  make  decisions. 
Required  Information:  Control  and  display  requirements, 

setup,  mode,  and  status  information. 

Required  oontroie:  Information  output  optimization 
controls. 

8.  Aircrew  function:  Manage  onboard  systems. 

Required  Information:  Requirements,  settings,  modes, 

and  status  of  all  systems. 

Required  controie:  System  setting  and  mode  optimization 
controls. 

9.  Aircrew  function:  Monitor  and  Interpret  system-provided 
information  and  make  decisions. 

Required  Information:  (a)  Current  setup,  mode,  and  status 
information  for  all  systems,  (b)  Mission  and  tactics 
information  related  to  each  system,  (c)  Data,  provided  by 
each  onboard  system. 

Required  oontroie:  System  output  optimization  controls. 

10.  Aircrew  function:  Recognize  and  overcome  system 
problems. 

Required  Information:  Warnings  about  system  problems. 
Required  controie:  Controls  for  recovery  from  system 
problems. 


Figure  3.  Top-Level  Listing  of  Aircrew  Functions  and 
Information  and  Control  Requirements. 


5.  Vigilance  and  fatigue.  In  its  surveillance  role,  the 
aircrew  will  be  required  to  spend  many  hours  seeking  and 
evaluating  potential  targets.  System  moding  and  display 
formats  should  be  optimized  to  yield  satisfactory  target 
acquisition  performance.  System  architectures  should 
support  the  pilot’s  needs  for  stimulation  and  interesting 
tasks,  critical  for  maintenance  of  a  vigilant  state  (Ref.  7). 

6.  Task  management.  In  the  attack  role,  target 
acquisition  and  precise  completion  of  tactical  maneuvers 
requires  system  architecture  that  includes  intelligent 
decision  support  (Ref.  5)  and  adaptive  automation  (Ref.  6) 
of  some  control  functions. 

Several  more  general  considerations  also  should  be 
considered  during  development  of  avionics  system 
architectures  for  advanced  fighters.  These  considerations 
include  the  following. 

1.  Systems  approach.  From  the  aircrew’s  perspective,  all 
components  of  an  advanced  fighter  are  part  of  a  single 
system:  the  aircraft.  For  satisfactory  crew  performance,  the 
aircraft  and  all  its  subsystems  must  incorporate  a  single, 
consistent  control  and  display  philosophy  (Ref.  8).  That  is, 
display  formats,  abbreviations  and  acronyms,  symbol 
meanings,  information  display  techniques,  control 
assignments  and  operations,  procedures,  and  relative 
degrees  of  automation  should  be  consistent  throughout 
(Ref.  6). 

2.  Layered  interface.  Over  time,  aircrews  with  various 
levels  of  sophistication  will  use  the  advanced  fighter  system. 
The  architecture  should  provide  novice  users  with  a  menu- 
driven  interface,  but  permit  experienced  users  to  circumvent 
the  menus  and  give  direct  commands  to  the  system  for  rapid 
operation  (Ref.  9).  “Deep  windowing”  is  bad;  people  get 
lost  when  they  must  navigate  through  numerous  menu 
levels. 

3.  Separation  of  user  interface  code.  The  architecture 
should  be  designed  to  separate  the  programs  and  code  that 
determine  specific  display  and  control  operations  (the  user 
interface)  from  the  programs  that  actually  operate  the 
systems.  This  is  important  to  facilitate  rapid  and  inexpensive 
improvements  in  the  user  interface,  as  aircrew  personnel 
evaluate  the  system  and  propose  changes  (Ref.  10). 

4.  Logical  command  and  option  ordering.  Commands 
and  options  should  form  logical  and  consistent  sets.  The 
command  functions  and  the  names  given  to  commands 
should  be  predictable  by  the  aircrew  operators  (Ref.  8). 

5.  Minimal  aircrew  operations.  System  control  actions 
carried  out  the  by  the  aircrew  must  be  simple,  logical,  and 
consistent,  yet  flexible  enough  that  the  aircrew  may  tailor 
the  order  in  which  actions  are  completed  both  to  the 
mission  and  for  personal  preferences. 

6.  No  “garden  pathing  ”  Under  no  circumstances  should 
the  architecture  allow  the  aircrew  to  continue  data  entiy  or 
other  system  operations  after  an  entry  has  resulted  in  an 
error  condition  that  will  nullify  the  entire  procedure.  Error 
checking  should  be  continuous.  The  aircrew  should  be 


notified  immediately  if  a  disallowed  action  has  been  taken, 
for  immediate  correction  without  loss  of  other  operator 
inputs  (Ref.  9). 

7.  One-time  data  entry.  Under  no  circumstances  should 
the  aircrew  be  required  to  enter  the  same  data  more  than 
once.  All  systems  requiring  data  should  be  able  to  access  a 
single  source  and  obtain  whatever  is  needed  from  a  single 
(usually  premission)  aircrew-entered  data  file  (Ref.  8). 

8.  System  response  time.  Aircrew  commands,  menu 
selections,  and  data  entry  actions  must  be  acknowledged 
rapidly,  delays  should  not  exceed  0.2  second  (Ref.  1 1).  If 
the  aircrew  must  make  a  real-time  response,  the  systems 
must  provide  a  satisfactory  answer  in  the  available  time, 
with  solution  quality  improving  monotonically  with  time 
(Ref.  12). 

9.  Aircrew  trust.  The  advanced  fighter  system 
architecture  should  ensure  that  all  aircrew-delegated, 
mission-critical  functions  are  performed  reliably  by  the 
systems  to  which  they  are  delegated  (Ref.  9).  The  systems 
should  recommend  intelligent  options,  and  exercise  the  level 
of  autonomy  conferred  by  the  aircrew.  The  aircrew  remains 
the  ultimate  source  of  mission  priorities  and  a  vital  source  of 
status  and  situation  information. 

&2  Aircraft  Systems 

For  this  study,  aircraft  systems  are  defined  as  those  used  by 
the  aircrew  for  monitoring  and  controlling  aircraft  flight. 
These  systems  include  the  mission  and  aircraft  computers; 
engines;  aircraft  control  surfaces;  fuel  systems;  and 
monitoring  systems  related  to  attitude,  altitude,  airspeed, 
and  direction  of  flight.  Aircraft  systems  should  provide  the 
required  airspeeds,  fuel  capacities,  and  other  performance 
and  operational  capabilities  needed  for  various  operational 
roles,  yet  be  compatible  with  avionics  and  fire  control 
systems,  with  missile  system  payloads,  and  with  aircrew 
capabilities. 

Human  capabilities  and  limitations  related  to  aircraft  system 
functions  are  discussed  below. 

1.  Spatial  disorientation  and  vertigo.  Various  flight 
conditions,  including  the  use  of  night  vision  goggles  (NVGs) 
can  result  in  the  aircrew  becoming  disoriented  regarding 
their  position  or  orientation  with  respect  to  the  earth  (Ref. 
13, 14).  Virtually  all  pilots  experience  this  at  some  time 
(Ref.  13).  Aircraft  systems  should  be  designed  to  minimize 
procedures  that  result  in  disorientation,  to  aid  in  recovery 
from  unusual  attitudes,  and  to  provide  ground  proximity 
warnings. 

2.  Continuous  control  Continuous  closed-loop  control 
tasks  required  for  aircraft  flight  control  and  navigation  are 
extremely  demanding  for  the  aircrew  (Ref.  16).  These  tasks 
form  a  workload  baseline  on  which  other,  noncontinuous 
aircrew  tasks  build.  Thus  workload  minimization  should 
begin  with  making  basic  flight  control  procedures  as  simple 
and  intuitive  as  possible.  Careful  automation  and  the  use  of 
artificial  intelligence  techniques  should  be  considered. 


3.  G-induced  loss  of  consciousness.  Increasing  aircraft 
speed,  acceleration,  and  turning  abilities  beyond  those 
tolerated  by  humans  is  definitely  counterproductive. 
Crewstation  design  should  minimize  the  probability  that  the 
aircrew  will  suffer  degraded  visual  acuity,  reduced  ability  to 
do  mental  processing,  and  GLOC  (Ref.  17).  When 
conditions  warrant,  the  ability  should  be  provided  for  the 
aircrew  to  turn  some  critical  functions  and  procedures  over 
to  automated  systems.  In  some  situations,  the  ability  of 
aircraft  systems  to  sense  that  the  crew  has  been 
incapacitated  will  be  necessary  so  that  automatic  control  can 
be  initiated. 

4.  Eye  damage.  The  human  visual  system  is  extremely 
delicate  and  subject  to  temporary  or  permanent  damage 
from  brilliant  flashes  and  from  lasers.  Some  instantaneous 
procedure  (such  as  making  the  helmet  visor  or  canopy 
opaque)  should  be  provided  to  protect  aircrew  eyes  from 
such  dangers. 

5.3  Control  and  Display  Systems 
For  this  study,  control  and  display  systems  in  the  crewstation 
are  those  devices  used  by  the  aircrew  (1)  to  give  commands 
to  the  aircraft  and  its  systems  (usually  indirectly,  through  a 
computer  system),  and  (2)  to  obtain  information  from  the 
aircraft  and  its  systems  (again,  usually  with  a  computer 
system  serving  as  intermediary).  Crewstation  control  and 
display  systems  should  enable  the  aircrew  to  perform 
control  actions  and  to  obtain  information  as  needed  to 
manage  the  aircraft  and  weapons  systems.  Task  assistance 
systems  are  needed,  including  various  aircrew-selected  levels 
of  automation  as  appropriate  for  the  operational  situation. 
Decision-aiding  systems  also  are  required,  possibly  including 
expert  systems. 

Aircraft  control  systems  include  the  traditional  throttle  and 
control  stick,  plus  a  selection  of  pushbuttons,  toggle 
switches,  and  joysticks  located  on  the  throttle  and  stick  (the 
“hands-on-throttle-and-stick”  or  HOTAS  control 
philosophy).  Some  control  operations  are  provided  via 
dedicated  switches  (e.g.,  weapon  jettison  and  “Master  Arm” 
selections).  However,  as  aircrew  functions  have  proliferated, 
multipurpose  controls  are  used  where  practical  to  minimize 
crewstation  crowding  and  clutter.  Multipurpose 
pushbuttons  often  are  integral  to  multifunction  displays,  so 
that  part  of  the  display  surface  can  be  used  to  label  the 
switches  with  their  current  functions.  Other  multipurpose 
pushbuttons  are  part  of  control  systems  that  include  a  10- 
digit  keypad  for  entry  of  latitude/longitude,  range,  and  other 
numeric  data. 

Aircraft  display  systems  rely  increasingly  on  presenting 
critical  information  “head  up”  so  the  aircrew  does  not  have 
to  look  down  into  the  cockpit  and  search  for  needed  data. 
The  head-up  display  (HUD)  currently  is  the  primary  device 
in  this  category,  providing  flight,  navigation,  and  targeting 
data  when  the  pilot  is  looking  forward  through  his 
windscreen.  However,  advanced  fighter  display  concepts 
also  should  include  helmet-mounted  displays  (HMDs)  that 
can  provide  similar  data  wherever  the  pilot  is  looking  (Ref. 
18).  Since  clutter  and  the  danger  of  distractions  prevent 
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displaying  more  than  the  minimum  critical  information  head 
up,  "head  down*  multifunction  displays  also  are  needed  to 
provide  detailed  information  about  aircraft,  weapon,  target, 
and  threat  status.  Crews tation  research  also  includes 
concepts  such  as  the  ‘big  picture”  cockpit  where  a  single  flat 
panel  or  high-resolution  TV  display  fills  the  main 
instrument  panel  and  replaces  all  the  individual  display 
devices  with  integrated  presentation  of  information  as  it  is 
required  (Ref.  19).  “Virtual  cockpits”  generated  using 
HMDs  or  hologram  technology  to  present  “controls  and 
displays  in  the  air*  also  are  being  explored  (Ref.  20). 

Consideration  of  various  modern  crews  tation  control  and 
display  concepts  for  advanced  fighter  applications  is 
important.  However,  the  ability  of  the  aircrew  to  perform  all 
mission-required  tasks  using  many  of  these  advanced 
concepts  is  largely  untested  at  present. 

Human  capabilities  and  limitations  related  to  the  operation 
and  use  of  controls  and  displays  are  discussed  below. 

1.  Individual  differences.  Humans  vary  widely  in  their 
cognitive,  physiological,  and  physical  requirements  related 
to  the  use  of  control  and  display  systems.  As  a  result,  several 
factors  will  strongly  affect  aircrew  ability  to  carry  out  the 
advanced  fighter  missions  satisfactorily.  These  include  the 
individual’s  (a)  prior  training  and  aircraft-related 
experiences,  (b)  peripheral  vision  field  size,  (c)  auditory 
decrement  at  some  frequencies  due  to  noise  exposure,  (d) 
ease  of  distraction,  (e)  ability  to  maintain  a  vigilant  state,  (f) 
reaction  to  stress,  (g)  cerebral  hemisphere  preference 
(whether  he  demonstrates  primarily  a  spatial  or  verbal 
orientation),  (h)  locus  of  control  (internal  or  external 
motivation),  (i)  risk  aversiveness  level,  (j)  willingness  to  yield 
control  to  his  systems,  (k)  decision  response  time,  and  (I, 
control  reaction  time  (Ref.  15).  In  the  past,  crewstation 
designers  have  ignored  individual  differences.  Controls  and 
displays  have  been  optimized  for  the  “average”  or  “median” 
aircrew.  It  now  is  passible  to  design  aircraft  displays  and 
controls  so  that  each  crewmember  can  tailor  to  his  personal 
needs  what  he  sees  and  hears  and  how  he  controls  his 
systems.  For  optimum  aircrew  performance  of  the  advanced 
fighter  mission,  this  approach  definitely  should  be 
considered  for  advanced  fighter  development. 

2.  Empty  field  myopia  and  accommodation-convergence 
micropsia.  When  looking  at  HUD  and  HMD  symbols 
against  the  sky,  for  some  people  the  eye  tends  to  focus  at 
about  2  feet  out  (even  though  HUDs  and  HMDs 
supposedly  are  focused  at  infinity).  This  results  in 
misjudgments  of  the  sizes  of  and  distances  to  external 
objects  (such  as  other  aircraft),  when  these  come  into  view 
(Ref.  21).  These  problems  are  aggravated  by  the  use  of 
NVGs.  Aircraft  systems  should  be  designed  to  minimize 
visual  illusions. 

3.  Head-up  missions.  Even  with  modem  sensors,  the 
fighter  role  requires  that  the  aircrew  be  able  to  focus 
attention  “out  the  window”  during  the  heat  of  battle.  The 
importance  of  head-up  displays  and  controls  to  satisfactory 


mission  accomplishment  cannot  be  overemphasized.  Thus 
during  design  of  the  advanced  fighter  crewstation  major 
attention  should  be  paid  to  optimum  integration  of  head-up 
and  helmet-mounted  displays,  NVGs,  and  auditory  cueing, 
along  with  that  data  which  still  must  be  presented  head 
down.  Control  systems  should  rely  strongly  on  voice 
commands  (Ref.  22)  and  on  head-  and  eye-tracking  devices; 
requirements  to  use  manual  controls  should  be  minimized. 

4.  Environmental  protection  gear.  Aircrews  operate 

aircraft  controls  and  view  displays  while  tightly  strapped  into 
an  ejection  seat  and  wearing  restrictive  clothing  and  gloves. 
More  recently,  the  crew  also  must  be  able  to  obtain  system 
information  and  to  control  all  of  their  systems  while 
protected  by  cumbersome  chemical  and  biological  warfare 
gear.  All  required  aircrew  personal  gear  should  be 
predefined  and  considered  during  development  of  the 
crewstation,  for  satisfactory  advanced  fighter  mission 
accomplishment. 

5.4  Navigation  Systems 

For  this  study,  navigation  systems  are  defined  as  equipment 
used  by  the  aircrew  to  monitor  and  control  aircraft  position 
and  direction  of  flight.  These  systems  include  inertial 
navigation  systems,  compasses,  digitized  maps,  and  signals 
from  global  positioning  satellites.  They  also  include 
computer  systems  that  store  data  related  to  the  mission 
plan,  flight  path,  event  timing,  and  tactics.  Navigation 
systems  must  be  capable  of  navigating  to  any  point  by  way  of 
a  predetermined  course,  must  provide  position  (latitude  and 
longitude)  and  velocity  vector  information  continuously,  and 
must  support  cooperative  engagements. 

Human  capabilities  and  limitations  related  to  navigation 
system  functions  are  discussed  below. 

1.  Ability  to  replan.  When  unexpected  events  result  in 
unforeseen  situations,  humans  generally  are  very  good  at 
reprioritizing  and  replanning  to  accommodate  to  the 
situation.  However,  humans  are  not  good  at  mental 
calculations  or  holding  more  than  five  to  nine  items  in  mind 
at  one  time.  Thus,  the  ability  to  replan  a  mission  in  the  air  if 
required  should  be  augmented  by  computer  aids  that  can 
assist  with  the  required  replanning  tasks. 

2.  Decision  making.  Humans  primarily  reason  from 
examples,  based  on  what  has  worked  for  them  before  and 
what  they  can  recall  easily.  They  rarely  can  consider  more 
than  three  or  four  hypotheses  at  one  time.  They  refuse  to 
give  up  a  theory  or  model  unless  they  have  another  with 
which  to  replace  it.  They  also  reason  using  backward 
chaining,  choosing  a  goal  or  solution  then  seeking  a  way  to 
reach  that  goal  (Ref.  23).  Thus,  when  navigation  to  or  from 
the  target  area  requires  that  the  aircrew  make  decisions, 
computer  assistance  should  be  provided  (a)  to  propose  the 
most  reasonable  alternatives  and  hypotheses,  (b)  to  help  the 
aircrew  visualize  these  alternatives  and  objectively  weigh 
their  pros  and  cons,  and  (c)  to  provide  the  sequence  of  steps 
necessary  to  reach  the  solution  or  desired  goal  (Ref.  24). 
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SJS  Communications  Systems 
For  this  study,  communications  systems  are  defined  as  those 
onboard  systems  used  by  the  aircrew  to  communicate 
between  themselves  and  with  surface,  subsurface,  ground- 
based,  and  other  airborne  elements,  including  both  manned 
stations  and  automatic  datalink  facilities.  Communications 
systems  must  transmit  and  receive  jam-free  data 
continuously  to  and  from  any  of  the  force  structure 
elements  employed  in  the  specific  tactical  operations, 
including  voice,  datalink,  and  missile  command  guidance 
communications. 

Human  capabilities  and  limitations  related  to 
communications  and  communication  systems  are  discussed 
below. 

1.  Information  fusion.  During  the  mission,  the  aircrew 
must  utilize  real-time  voice  and  datalinked  information  in 
conjunction  with  intelligence,  geographic,  and  cultural  data 
entered  into  the  system  during  premission  planning. 
Humans  have  a  tendency  to  give  the  greatest  weight  to 
earlier  data,  using  recent  data  primarily  to  seek 
confirmation  for  that  early  data  rather  than  to  test  it  (Ref. 
23).  Techniques  are  needed  to  integrate  data  from  real-time 
communications  systems  with  premission  data,  to  provide 
the  aircrew  with  the  best  possible  objective,  unbiased  “world 
picture”  as  that  picture  changes  dynamically. 

2.  Information  uncertainty.  Humans  have  a  tendency  to 
treat  all  information  as  if  it  were  equally  reliable.  They 
generally  are  not  good  at  estimating  probabilities,  or  at 
weighing  and  comparing  the  relative  “goodness”  of  data 
from  various  sources.  They  are  especially  bad  at  combining 
probabilities  and  at  including  a  priori  probabilities  in  their 
mental  calculations  (Ref.  23).  The  advanced  fighter  systems 
should  provide  the  aircrew  with  assistance  in  evaluating  the 
level  of  certainty  for  information  communicated  from 
various  individual  sources,  plus  the  level  of  certainty  for 
integrated  information  from  several  sources. 

5.6  Sensor  Systems 

For  this  study,  sensor  systems  are  defined  as  those  onboard 
avionics  systems  that  provide  real-time  data  to  the  aircrew 
about  targets  and  some  kinds  of  threats.  Sensors  include 
both  passive  and  active  systems.  They  may  be  based  on 
radar,  television,  infrared,  radio  and  millimeter  wave,  and 
other  technologies.  Sensor  systems  must  be  sensitive 
enough  to  provide  the  characteristics  of  anticipated  targets 
in  various  operational  electromagnetic  and  climatic 
environments. 

Human  capabilities  and  limitations  related  to  the  use  of 
sensor  systems  are  discussed  below. 

1.  Monitoring  and  vigilance.  The  human  generally  is 
easily  distracted  arid  is  a  very  poor  system  monitor  (Ref  15). 
For  satisfactory  mission  performance,  some  level  of 
automation  probably  will  be  needed  for  monitoring  sensor 
output  for  prespecified  events  that  occur  infrequently  and 
unexpectedly,  so  that  the  aircrew  can  be  alerted. 


2.  Mental  processing  of  information.  A  great  deal  of 
research  has  been  done  on  the  best  ways  to  display  both 
sensor  images  and  computer-generated  formats  based  on 
sensor  outputs,  for  satisfactory  aircrew  understanding  and 
performance.  Appropriate  handbooks,  sets  of  guidelines, 
and  related  materials  should  be  used  in  design  of  the 
advanced  fighter  sensor  output  displays  to  ensure  that 
symbols  and  formats  are  used  which  have  been  tested  and 
evaluated  (Ref.  8). 

3.  Operational  decision  making  If  sensor  image  quality  is 
uncertain,  the  aircrew  will  have  a  hard  time  deciding 
whether  to  trust  what  is  displayed.  Providing  an  associated 
image  quality  metric  that  indicates  the  current  relative 
“goodness”  of  sensor  output  will  result  in  better  targeting 
decisions. 

5.7.  Targeting  Systems 

For  this  study,  targeting  systems  are  defined  as  those 
onboard  sensor  and  computer  systems  that  assist  the 
aircrew  in  detecting,  identifying  (friendly,  unknown,  hostile), 
classifying  (specific  target  type),  and  designating  airborne, 
surface,  and  ground  targets.  The  classification  abilities  of 
onboard  targeting  systems  may  be  enhanced  by  computer- 
based  premission  data  related  to  anticipated  targets  and 
various  characteristics  of  these  targets,  and  by  up-to-date 
datalinked  information  from  other  airborne  platforms  and 
from  surface  assets.  It  should  be  noted  that  basic  target 
aircrew  functions  related  to  targeting  system  control  and 
information  interpretation  are  similar  to  those  required  for 
the  use  of  aircraft  sensor  systems  in  general.  Target 
detection  and  classification  systems  must  detect,  identify, 
and  classify  targets  at  maximum  surveillance  and 
engagement  acquisition  ranges. 

Human  capabilities  and  limitations  related  to  targeting 
systems  are  discussed  below. 

1 .  Information  gathering  During  decision  making, 
humans  tend  to  seek  far  more  information  than  they  can 
easily  absorb,  especially  if  decisions  are  hard  or  outcomes 
expensive  (as  is  the  case  during  target  classification)  (Ref. 
25).  As  information  is  gathered,  humans  become  more 
confident  in  their  decisions,  but  not  necessarily  more  correct 
in  their  decisions  (Ref.  23). 

2.  Anchored  judgments.  While  considering  alternatives, 
undue  weighting  is  given  to  early  evidence,  which  anchors 
the  judgment  at  a  starting  value  (Ref.  26).  Then  humans 
tend  to  seek  (and  usually  find)  additional  information  that 
confirms  the  decision  or  chosen  course  of  action,  rather 
than  to  test  it;  they  exhibit  “cognitive  tunnel  vision”  (Ref. 
25).  This  is  especially  true  under  conditions  of  high  stress 
and  workload  such  as  when  deciding  whether  an  object  is  to 
be  attacked.  Humans  will  not  abandon  a  hypothesis  or 
theory  (no  matter  how  bad)  unless  they  have  another  with 
which  to  replace  it  (Ref.  23). 

3.  Limited  consideration  of  sources  and  alternatives. 
Limitations  of  human  attention  and  working  memory  make 
it  impossible  to  integrate  information  simultaneously  from 
more  than  a  few  sources  (Ref.  25).  This  limitation  can  affect 
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whether  an  adversary  is  identified  in  time  for  first  launch  of 
an  air-to-air  missile.  In  addition,  during  decision  making 
humans  tend  to  focus  on  a  few  critical  attributes  at  a  time, 
and  consider  only  the  two  to  four  alternatives  or  options 
that  rank  highest  on  those  particular  attributes  (Ref.  23). 

4.  Conservative  judgments.  Humans  generally  are  ‘risk 
aversive*  and  exhibit  a  central  tendency  of  odds;  extreme 
values  are  not  given  as  much  confidence  as  might  be 
optimum  for  reliably  determining  the  nature  of  a  possible 
target  (Ref.  25). 

SA  Weapons  Interface  and  Control  Systems 
For  this  study,  weapons  interface  and  control  systems  are 
defined  as  those  onboard  systems  used  for  weapon  carriage, 
to  initialize  and  prepare  weapons  for  release,  and  to  control 
inflight  weapons.  Computer  systems  required  for  specific 
weapons  also  are  included  in  this  definition.  Weapons 
interface  and  control  systems  must  be  capable  of  carrying  all 
required  weapons  and  of  programming  and  controlling 
those  weapons  during  carriage  and  in  flight. 

Human  capabilities  and  limitations  related  to  weapons 
interface  and  control  systems  are  discussed  below. 

1.  Top-level  weapon  moding.  Humans  are  especially 
error  prone  when  required  to  complete  a  number  of 
detailed  procedures  under  time  stress  (Ret  27).  To 
minimize  this  effect,  the  weapons  systems  should  self- 
initialize  to  near-optimum  states  for  a  given  mission  and 
mission  phase.  For  example,  one-switch  selection  of ‘air-to- 
air  mode*  should  configure  the  stores  management  systems 
for  air  combat. 

2.  Adaptive  automation.  Weapon  delivery  is  workload 
intensive,  and  automation  of  some  functions  will  be 
necessary.  Automation  levels  can  be  adapted  to  the  mission 
and  situation  in  three  ways;  (a)  through  fixed  automation 
levels,  predetermined  by  the  aircraft  development  team, 
that  are  considered  appropriate  for  a  given  mission  phase, 
task,  and  situation;  (b)  via  aircrew  selection  and  tailoring  of 
task  automation  levels  prim-  to  each  mission,  to  suit  personal 
styles  (Ref.  28);  and  (c)  through  the  use  of  expert  systems 
or  neural  network  systems  that  determine  the  aircrew’s 
“intent”  based  on  the  current  situation,  and  select 
appropriate  automation  levels  (Ref.  29).  The  proper  use 
and  mix  of  these  techniques  will  be  important  for  aircrew 
performance. 

3.  Remote  vehicle  control  Aircrew  datalink  control  of  an 
inflight  missile  is  similar  to  control  of  any  remotely-piloted 
vehicle.  Design  of  such  systems  so  that  human  performance 
will  be  satisfactory  is  not  easy,  especially  for  the  terminal 
phase  of  weapon  delivery.  Existing  guidelines  should  be 
heeded  (Ref.  11). 


5.9.  Electronic  Warfare  Systems 
For  this  study,  electronic  warfare  (EW)  systems  are  defined 
as  those  onboard  systems  related  to  electromagnetic  (radio 
frequency  and  electro-optic)  signal  exploitation  and 
electronic  warfare  technologies.  These  include  electronic 
surveillance  devices;  radar  warning  receivers;  signal 
repeaters  and  jammers;  expendable  countermeasures  stores 
such  as  chaff,  flares,  and  minijammers;  and  missile  detection 
systems.  It  should  be  noted  that  basic  aircrew  functions 
related  to  EW  system  control  and  information 
interpretation  are  similar  to  those  required  for  the  use  of 
aircraft  sensor  systems  in  general.  EW  systems  must  detect, 
analyze,  and  assess  all  electromagnetic  signals  to  determine 
bearing,  category,  and  number  of  emitters;  provide 
automatic  and/or  manual  control  for  management  of 
offensive  and  defensive  EW  capabilities;  and  support 
targeting  and  missile  system  fire  control  functions. 

Human  capabilities  and  limitations  related  to  EW  systems 
are  discussed  below. 

1.  Information  interpretation.  In  high-stress  situations 
such  as  the  presence  of  missile  threats  (and  especially  when 
a  missile-in-the-air  alert  has  been  received),  human  mental 
functioning  is  undependable.  Displayed  information  should 
rely  on  simple,  intuitive  diagrams,  rather  than  words  or 
other  information  presentation  modes  that  require 
interpretation.  Information  should  be  provided  in  the  form 
of  commands — directing  the  crew  precisely  what  to  do,  and 
when  to  do  it  (Ref.  15). 

2.  Memory  fallibility.  Short-term  or  working  memory  is 
seriously  hampered  by  stress.  The  ability  of  the  aircrew  to 
recall  required  procedures  cannot  be  depended  on.  Cues 
should  be  provided  to  help  the  crew  recognize  necessary 
sequential  actions. 

3.  Visual  system  overload.  The  aircrew  must  keep  track 
of  a  great  deal  of  information  needed  for  satisfactory 
situation  assessment  while  trying  to  complete  a  mission 
under  high-threat  conditions.  Most  of  this  information 
usually  is  received  visually.  Humans  are  capable  of 
processing  more  information  if  it  is  received  using  several 
sensory  systems  (e.g.,  both  visual  and  auditory  signals)  and  if 
the  modality  of  the  signal  is  compatible  with  normal  mental 
processing  of  the  particular  type  of  information  (e.g.,  spatial 
or  verbal)  (Ref.  25).  Ways  to  reduce  visual  overload  should 
be  sought,  with  consideration  of  substituting  auditory  signals 
to  represent  appropriate  kinds  of  information. 

4.  Control  of  self-protect  equipment  To  reduce  workload 
and  the  necessity  of  making  decisions,  aircrews  should  have 
the  option  of  commanding  automatic  signal  jamming  and/or 
the  dispensing  of  countermeasures  when  appropriate.  They 
also  must  be  able  to  countermand  such  automation  and 
assume  manual  control  as  desired.  As  with  the  visual 
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system,  the  aircrew’s  manual  control  abilities  are  near  their 
limits.  Techniques  are  needed  to  permit  system  control 
using  other  modalities,  including  voice  commands  and 
commands  given  via  head  or  eye  movements. 

6.  HUMAN  FACTORS  PROBLEM  AREAS 
While  a  list  of  aviation-related  human  capabilities  and 
limitations  is  interesting,  it  is  not  especially  useful  in  laying 
out  needed  research  programs  related  to  aircrew-aircraft 
interfaces.  Where  precisely  should  we  focus  our  efforts?  To 
answer  this  question,  the  40-odd  items  listed  above  have 
been  reviewed,  organized,  and  consolidated  into  28  problem 
areas  related  to  human  factors  engineering.  These  are  areas 
where  human  factors  engineers  currently  lack  some  of  the 
information  essential  for  development  of  crews tations  for 
advanced  military  fighter  and  attack  aircraft.  What  is 
needed  (at  least  in  part)  to  solve  each  of  the  problems  also 
has  been  identified  and  is  documented  below. 

The  28  human  factors  issues  can  be  separated  into  nine 
categories,  as  shown  in  Fig.  4.  Several  of  these  categories 
were  included  in  the  FY-1991  U.S.  Department  of  Defense 
Critical  Technologies  Plan  (Ref.  30).  The  DoD’s  report 
documents  20  technologies  considered  to  be  the  most 
important  to  ensuring  the  long-term  qualitative  superiority 
of  United  States  weapons  systems.  These  critical 
technologies  include  (1)  machine  intelligence  and  robotics, 
(2)  simulation  and  modeling,  (3)  signal  processing,  and  (4) 
data  fusion. 

6.1  Physical  and  Physiological  Stress. 

1.  Problem:  Environmental  protection  gear.  Needed: 
Gear  that  protects  from  biological  and  chemical  agents 
without  interfering  with  aircrew  tasks. 

2.  Problem:  G-induced  loss  of  consciousness.  Needed: 
(a)  Techniques  for  automatically  determining  when  GLOC 
is  impending,  (b)  Techniques  for  displaying  "GLOC 
impending"  warning  to  a  dazed  aviator,  (c)  Automatic 


aircraft  control  during  GLOC  episodes,  (d)  Techniques  for 
informing  the  crew  of  system  status  and  returning  control 
when  consciousness  is  regained. 

3.  Problem:  Laser  and  flash  protection.  Needed: 
Techniques  for  providing  eye  protection  that  do  not 
interfere  with  mission  tasks. 

6.2  Vigilance  and  Aircrew  Alerting 

4.  Problem:  " Situational  awareness.  ”  Needed:  Research 
to  demonstrate  techniques  and  procedures  that  result  in 
satisfactory  awareness  and  assessment  of  aircraft  and 
mission  status. 

5.  Problem:  Vigilance.  Needed:  (a)  Techniques  for 
monitoring  vigilance  and  alerting  the  crew  when  vigilance 
drops,  (b)  Techniques  for  directing  the  aircrew  to  take 
specific  emergency  actions,  (c)  Techniques  for  automating 
critical  monitoring  tasks  without  reducing  aircrew  vigilance 
levels,  (d)  Techniques  for  maintaining  vigilance  levels  on 
long  missions. 

6.  Problem:  Recovery  from  unusual  attitudes.  Needed: 
Research  on  sensory  modalities  and  display  techniques  for 
alerting  the  pilot  and  advising  how  to  return  to  a  safe  flight 
envelope. 

7.  Problem:  Ground  proximity  warnings.  Needed: 
Techniques  for  immediate,  intuitive  display  of  aircraft  data 
that  will  minimize  chances  of  flight  into  terrain. 

8.  Problem:  First  detection  and  launch.  Needed: 
Equipment  and  procedures  for  acquiring  air  adversaries  at 
maximum  range  and  alerting  the  crew  for  effective  weapon 
launch. 

6J  Individual  Differences 

9.  Problem:  Sensory  perception.  Needed:  (a)  Research  on 
performance  differences  with  various  information 
presentation  modalities  and  formats,  as  a  function  of  the 
individual  and  situation,  (b)  Techniques  for  aircrew  tailoring 
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Figure  4.  Categories  of  Problems  for  Advanced  Fighter  Aircrews  Related  to  Human  Factors. 
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of  display  modalities  and  formats  (both  premission  and 
midmission),  as  needs  and  conditions  vary. 

10.  Problem:  Mental  processing.  Needed:  Techniques  to 
ensure  that  crewstation  equipment  and  mission  procedures 
are  consistent  with  aircrew  expectations  and  mental  models. 

1 1.  Problem:  Response  and  control  Needed:  Techniques 
for  aircrew  tailoring  of  controls  and  operating  procedures 
(both  premission  and  midmission),  as  needs  and  conditions 
vary. 

6.4  Information  Integration 

12.  Problem:  Integrated  information  databases.  Needed: 
(a)  Techniques  to  permit  viewing  of  either  integrated, 
single-source,  or  multi-source  (crew-selected)  data,  as 
desired,  (b)  Procedures  and  guidelines  for  weighting  data 
according  to  uncertainty  levels  prior  to  integration,  (c) 
Techniques  that  advise  the  aircrew  of  the  sources  of  data 
and  that  permit  the  crew  to  evaluate  the  "goodness"  of  both 
integrated  and  unintegrated  data,  (d)  Techniques  that  result 
in  common,  consistent  symbols  and  formats  that  are 
intuitively  understandable,  regardless  of  the  information 
source  or  sources. 

13.  Problem:  Auditory  integrated  information  displays. 
Needed:  Research  on  the  best  uses  for  auditory  displays, 
best  types  of  auditory  signals  for  various  purposes,  and  most 
effective  vocal  frequencies,  words,  and  phrases  for  verbal 
displays,  under  all  mission  conditions. 

14.  Problem:  Visual  integrated  information  displays. 
Needed:  Procedures  and  guidelines  for  design  of  visual 
formats  presented  on  various  display  surfaces  (HUDs, 
HMDs,  etc.),  for  satisfactory  perception  and  understanding 
of  and  responses  to  integrated  data  under  all  mission 
conditions. 

15.  Problem:  Display  of  uncertain  information.  Needed: 
Techniques  for  displaying  uncertain  information  for 
intuitive  understanding  by  the  aircrew. 

6J5  Visual  Displays  for  Various  Missions 

16.  Problem:  Air  superiority  missions.  Needed:  (a) 
Research  on  HUD  and  HMD  information  display  to 
minimize  vertigo  and  disorientation,  and  to  counteract 
target  distance  and  size  illusions,  (b)  Research  on  HMD 
formats  that  can  be  used  effectively  during  air  combat 
missions. 

17.  Problem:  Surveillance  and  attack  missions.  Needed: 
(a)  Techniques  for  effectively  partitioning  and  integrating 
head-up  and  head-down  information,  (b)  Techniques  for 
effective  display  of  target  location,  offset  aimpoint,  and 
weapon  release-point  information,  (c)  Research  on 
minimizing  vertigo,  disorientation,  and  illusions  when  NVGs 
are  used,  (d)  Techniques  for  display  of  information  on 
NVGs  so  it  is  cognitively  compatible  with  other  displayed 
information. 

6.6  Mission  Management 

18.  Problem:  Mission  status.  Needed:  (a)  Techniques  to 
provide  meaningful  combinations  of  target  area,  target,  and 


weapon  information,  (b)  Techniques  to  provide  tactics- 
related  information  and  requirements  developed  during 
mission  planning,  (c)  Techniques  to  remind  the  crew  of 
preplanned  task  sequences  and  timing,  status  with  respect  to 
the  plans,  and  uncompleted  critical  tasks,  (d)  Techniques 
for  displaying  planned  versus  actual  aircraft  and  mission 
data  and  status,  (e)  Equipment  and  techniques  for  quick 
and  easy  partial  or  total  replanning  of  the  mission  during  the 
mission. 

19.  Problem:  System  status.  Needed:  Techniques  to 
inform  the  crew  of  available  system  modes,  current  system 
status,  and  system  interrelationships,  and  to  alert  the  crew 
to  system  problems. 

20.  Problem:  System  control  Needed:  (a)  Alternate 
techniques  for  system  control,  including  voice  commands, 
body  positioning,  head  or  eye  movements,  and  brain  activity, 
and  determination  of  the  kinds  of  tasks  for  which  each 
technique  is  satisfactory,  (b)  Research  on  the  maximum 
quantity  and  optimum  kinds  of  controls  for  HOTAS 
operations,  (c)  Techniques  for  simple  control  of  sensor 
setup  parameters  for  optimum  imagery,  (d)  Techniques  and 
procedures  for  simple  designation  and  undesignation  of  air 
and  surface  targets,  (e)  Techniques  for  optimum  display  of 
datalinked  information  from  inflight  weapons,  and  optimum 
techniques  for  controlling  weapons  in  flight. 

6.7  Decision  Support 

21.  Problem:  Decision  support  techniques.  Needed:  (a) 
Research  to  determine  and  prioritize  the  aircrew  tasks  for 
which  aiding  is  needed,  and  the  conditions  under  which  aids 
will  be  used,  (b)  Research  on  the  best  decision  support 
technique  for  each  kind  of  task,  (c)  Research  on  the  best 
aiding  medium  and  aid  development  technique  for  each 
kind  of  aid.  (d)  Research  on  the  most  appropriate  user 
interface  for  each  decision  support  system,  (e)  Techniques 
to  help  operators  integrate  information  from  a  variety  of 
sources,  overcome  typical  human  cognitive  biases,  and 
consider  and  weigh  all  reasonable  alternatives  when  making 
decisions. 

22.  Problem:  Decision  aid  usefulness.  Needed: 

Techniques  to  ensure  that  aids  can  be  trusted,  can  operate 
in  real  time,  and  can  justify  their  options  and  selections  for 
the  aircrew. 

6.8  Automation 

23.  Problem:  What  to  automate.  Needed:  (a)  Procedures 
and  guidelines  that  define  tedious  or  difficult  advanced 
fighter  tasks  that  are  suitable  for  automation,  (b) 

Procedures  and  guidelines  to  determine  the  optimum 
automation  level  and  technique  for  each  candidate  task. 

24.  Problem:  Adaptive  automation.  Needed:  (a)  Research 
to  determine  the  optimum  kind  and  level  of  adaptive 
automation  for  various  purposes,  (b)  Research  to  develop 
adaptive  automation  technologies,  (c)  Research  to  develop 
aircrew  procedures  for  use  of  adaptive  automation  systems, 
(d)  Research  to  determine  appropriate  display  formats  and 
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control  configurations  for  use  with  adaptive  automation 
systems. 

25.  Problem:  Overriding  automation.  Needed:  Techniques 
for  overriding  automated  systems  intuitively  and  rapidly  in 
emergencies. 

6.9  System  Design  and  Evaluations 

26.  Problem:  Specifications  and  statements  of  work. 
Needed:  Procedures  and  guidelines  to  ensure  that  advanced 
fighter  specifications  and  statements  of  work  require 
contractor  estimates,  demonstrations,  and  evaluations  of 
satisfactory  aircrew  performance  early  enough  to  affect 
designs. 

27.  Problem:  How  to  test  Heeded:  (a)  Procedures  and 
guidelines  for  optimum  allocation  of  mission  functions 
among  aircrew  personnel  and  between  the  aircrew  and 
aircraft  systems,  (b)  Procedures  and  guidelines  to  determine 
that  crewstation  equipment  is  optimum  for  the  missions,  (c) 
Procedures  and  guidelines  to  determine  that  crewstation 
equipment  layout  is  optimum  for  the  missions,  (d) 
Procedures  and  guidelines  to  ensure  that  required  aircrew 
operations  and  tasks  can  be  completed  in  accordance  with 
mission  requirements. 

28.  Problem:  How  to  evaluate.  Needed:  (a)  Procedures, 
techniques,  and  tools  to  specify  what  will  be  satisfactory 
levels  of  aircrew  performance,  (b)  Procedures,  techniques, 
and  tools  to  evaluate  whether  the  contractor’s  designs  will 
meet  specified  levels  of  aircrew  performance. 

7.  CONCLUSIONS  AND  RECOMMENDATIONS 
The  28  human  factors  problem  areas  discussed  above  may 
provide  a  near-term  basis  for  setting  some  of  our  human 
factors  research  and  development  priorities  related  to  the 
aircrew-aircraft  interface.  It  should  be  noted  that  these 
problems  are  not  listed  in  priority  order.  Although  each  type 
of  problem  is  critical  to  the  aircrew  in  some  manner,  further 
analysis  is  needed  to  determine  (1)  how  relatively  important 
this  problem  area  is  to  mission  success,  (2)  the  extent  of 
work  needed  to  address  each  problem,  and  (3)  the  order  in 
which  the  problem  areas  should  be  addressed  for  greatest 
efficiency  and  effectiveness. 

The  picture  is  not  entirely  gloomy.  A  great  deal  already  is 
widely  known  and  well  understood  about  how  to  optimize 
human  performance  under  various  conditions.  Some  of  that 
information  has  been  included  in  this  paper,  as  it  applies 
specifically  to  design  of  aircrew-aircraft  interfaces  for 
advanced  fighters.  Much  more  is  documented  in  widely 
available  books,  reports,  and  articles. 

The  above-noted  28  problem  areas  probably  cannot  all  be 
satisfactorily  addressed  in  this  decade.  However,  thoughtful 
collection  of  new  information  and  appropriate  use  of 
existing  human  factors  knowledge  will  significantly  increase 
the  chances  that  advanced  aircrew-aircraft  interfaces  will  be 
satisfactory  and  that  aircrews  will  be  able  to  carry  out 
constrained  and  limited  objective  naval  air  missions  needed 
in  the  early  21st  century. 
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1.  INTRODUCTION 

Modern  cockpit  designs  require  new  modular 
architectures  for  MISSION  and  IMAGE- 
MANAGEMENT  with  regards  to  hardware  and 
software  aspects. 

The  main  task  (see  figure:  1)  is  the  collection  of 
aircraft  specific  data  using  the  appropriate  DATA 
MANAGEMENT,  the  transformation  of  such  data  to 
graphical  images  with  the  appropriate  LOGICAL 
IMAGE  MANAGEMENT,  the  generation  of  physical 
graphical  images  on  several  image  devices  by 
PHYSICAL  IMAGE  MANAGEMENT  and  the 
conversion  and  combination/mixing  of  physical 
graphical  data  with  the  data,  created  by  external  video 
sensors  using  VIDEO  MANAGEMENT.  Finally  the 
video-data  has  to  be  presented  on  several  devices,  like 
HEAD  DOWN-,  HEAD  UP  and  HELMET 
MOUNTED  DISPLAYS. 

The  main  goal  for  us  as  basic  system  supplier  is  to 
give  the  application  programmer  an  abstract  high- 
level  interface  for  all  these  functions.  This  is  to  be 
done  and  is  specially  supported  by  die  program 
language  Ada,  which  is  the  required  language  for 
military  and  civil  aircraft  applications. 

The  system  described  herein  was  developed  for  the 
german  experimental  helicopter  program  AVT  and 
two  special  applications  for  the  X31A  experimental 
aircraft 


2,  SYSTEM  ARCHITECTURE 

The  system  architecture  (see  figure:  2)  consists  of 
several  modules  connected  to  a  system  bus,  which 
actually  is  the  VMEbus.  For  future  aspects  this  could 
also  be  the  PIbus  or  the  SAFEbus,  or  a  high  speed 
serial  data  busses  which  not  has  been  defined.  The 
system  bus  ensure  the  intermodule  communication 
and  must  support  multiprocessing  as  well  as  direct 
memory  access  (DMA)  and  fast  interrupt  capability. 

The  basic  architecture  connects  all  modules  via  a 
global  system  bus.  These  modules  are  data  interfaces, 
central  processing  units,  memory  modules,  graphic 


processing  modules  and  video  mixing  and  video 
conversion  modules. 

In  complex  systems  with  just  one  system  bus,  this 
one  bus  could  be  the  bottle  neck.  Because  of  this  a 
complex  system  may  require  more  than  just  one 
system  bus.  To  overcome  this  problem  we’ve 
designed  several  bus  structures. 

2.1.  Single  Bus  architecture 

When  the  data  transfer  rate  and  the  interrupt  load  of 
the  system  are  below  a  specific  value,  a  single  bus 
structure,  where  all  modules  are  connected  to  one  bus, 
will  be  sufficient  (see  figure:  3). 

2.2.  Separate  Bus  architecture 

In  this  solution,  some  modules  can  be  connected  to  an 
additional  bus  to  shift  the  data-transfer  load  from  the 
system  bus  to  this  additional  bus.  We  are  using  the 
VSBbus  in  addition  to  the  VMEbus.  That  VSBbus  is 
connected  parallel  to  the  VMEbus  over  the  same 
P1/P2  connector  (see  figure:  4). 

A  good  solution  could  be  in  connecting  CPUs  to 
special  data  or  graphic  processing  devices,  which 
require  a  more  complex  device  handling  and 
interaction. 

23.  Splitted  Bus 

This  solution  cuts  the  global  system  bus  in  halves, 
connected  via  a  DPR  memory  interface.  This  memory 
can  be  handled  fro m  both  sides  under  the  same 
address  and  supports  the  interprocessor 
communication  with  read-modify-write  capability. 
Two  CPUs  communicating  on  one  bus  side,  do  this  in 
the  same  way  as  on  both  sides  over  the  DPR  memory 
interface.  They  don’t  recognize  the  difference.  The 
system  can  grow  up  to  the  double  bus  load  with  the 
same  communication  facility  (see  figure:  3). 

2.4.  Mixed-Bus  architecture 

The  mixed  bus  structure  is  a  mixture  of  the 
SEPARATE  and  the  SPLITTED  bus  structure  and 
combines  their  advantages. 


3.  SYSTEM  COMPONENTS 

3.1.  Interfaces 

The  main  goal  here  is  to  collect  all  die  data  from 
different  links  and  make  it  available  to  the  system. 
This  has  to  be  done  by  local  device  intelligence  to 
minimize  the  handling  overhead  for  the  system  (see 
figure:  6).  Because  of  the  history  of  aircraft 
manufacturing,  many  different  data  links  exist 

For  future  aspects  the  collection  of  data  of  different 
links  and  the  transformation  to  one  modem  link  (e. 
AR INC-629)  could  be  implemented  in  separate 
cabinets. 

3.1.1.  Serial  interfaces 

Serial  links,  such  as  RS232,  RS422,  synchronous  or 
asynchronous  are  to  be  handled  with  local  device 
intelligence  to  fomj  single  data  items  into  complete 
data  sets  and  make  them  available  to  the  system. 
Because  the  devices  generate  an  interrupt  for  each 
single  data  item,  a  local  CPU  is  required  to  decrease 
the  interrupt  load  on  the  global  system  bus. 

3.1.2.  Parallel 

The  main  goal  here  is  to  start  and  stop  processes 
connected  to  single  inputs  and  produce  single  outputs. 
This  has  to  be  done  by  interrupt  generation  on  input 
value  changes,  asynchronous  reads  and  simple  writes. 

3.1.3.  MU-1553  interfaces 

When  operating  as  a  bus-controller  a  local 
intelligence,  which  supports  several  synchronous  data 
frames  with  predefined  error  handling  is  required.  For 
the  system  a  synchronous  and  an  asynchronous  data 
interface  are  required.  The  asynchronous  data 
interface  and  the  error  exceptions  have  to  be 
implemented  via  interrupts  to  support  short  latency 
and  determinisms. 

When  operating  as  a  remote-terminal  the  device 
handling  is  more  simple  and  will  be  covered  by  the 
devices  available  on  die  market  Local  intelligence  is 
only  required  when  special  synchronous  data  handling 
is  required. 


3.1.4.  Arinc-429  interfaces 

Synchronous  reads  are  supported  by  interrupts  and 
asynchronous  reads  are  supported  by  a  memory 
driven  interface.  Writing  data  are  supported  by  special 
FIFO-buffers,  where  the  data  can  be  filled  in  at  the 
maximum  bus-speed  by  the  system. 

3.1.5.  Arinc-629  interfaces 

Synchronous  reads  are  supported  by  interrupts  and 
asynchronous  reads  are  supported  by  a  memory 
driven  interface.  Writing  data  is  supported  with 
special  page-buffers,  where  the  data  can  be  filled  with 
bus-speed  by  the  system.  The  data  transfer,  the 
protocol  and  the  framing  are  supported  directly  by  the 
hardware. 

Hie  use  of  a  special  cabinet,  to  collect  data  from 
different  data  links  and  to  convert  it  into  one  high 
speed-bus  like  ARINC-629  will  result  in  a  reduction 
the  number  of  interfaces  to  just  one  (  dual  or  triple 
redundant)  ARINC-629  interface  within  this  system. 

3.2.  Central  processing  modules 

Two  major  solutions  are  available:  CISC  and  RISC. 
The  reduced  instruction  set  processors  are  designed 
for  more  computing  power  than  complex  instruction 
set  processors.  However,  the  CISC  designers  try  to 
stay  in  the  race  by  issuing  the  same  computing 
power,  as  in  the  RISC  market.  In  the  area  of  20 
MIPS,  CISC  and  RISC  CPUs  are  available  (  e. 
68040, 80486, 88k,  spare ).  The  next  RISC  generation 
of  about  100  MIPS  is  available  (  a.e.  88110  )  and  we 
are  waiting  for  the  CISC  answer  (see  figure:  7). 

However,  there  is  another  point  in  this  discussion 
which  is  important  for  aircraft  processing  systems. 
The  RISC  code  requires  2  to  3  times  more  memory 
than  the  CISC  code.  Our  decision  in  this  discussion 
was  to  stay  with  the  CISC  processors,  as  long  as  they 
have  the  same  range  of  computing  power  as  the  RISC 
processors.  Because  of  the  higher  memory 
requirements  in  the  RISC  area,  we  will  switch  over 
when  this  solution  brings  much  more  computing 
power  then  the  CISC  solution.  This  could  be  done  by 
placing  up  to  4  RISC  processors,  each  performing 
100  MIPS,  on  one  board  (e.  4  *  881 10). 
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For  our  system,  both  CISC  and  RISC  solutions  are 
available. 

33.  Global  memory  modules 

The  global  system  memory  is  required  for  inter 
processor  communication  and  for  storing  the 
processor  code. 

33.1.  Communication  memory  module 

This  memory  module  supports  the  inteiprocessor 
communication.  This  has  to  be  done  by  working  with 
read-modify-write  cycles.  This  memory  could  be 
realized  by  an  additional  memory  board,  or  by 
exporting  local  cpu  memory  to  the  global  system  bus. 

33.2.  Recourse  memory  module 

This  memory  module  stores  the  system  code  for  all 
processors  and  is  realized  by  special  EEPROM 
devices.  With  power-on  each  CPU  loads  its  code 
from  that  memory  into  its  local  memory  and  executes 
it.  Executing  the  code  direct  from  this  kind  of  devices 
is  not  effective  because  of  the  longer  access  times. 
Because  of  the  online  programmability  of  this 
memory,  software  updates  in  the  system  lifetime  are 
very  easy. 

3.4.  Graphic  processors  modules 

Graphic  processors  must  have  special  local 
intelligence  to  transform  graphical  commands  to 
screen  oriented  objects.  Because  of  more  growing 
graphical  power  in  2D-  and  3D  aspects  we  decided  on 
a  general  purpose  architecture  (Texas  34020). 
Graphical  transformations  are  specially  supported  with 
a  highspeed  (  20  MFLOPS  )  floatingpoint  processor, 
which  can  act  fully  parallel  to  the  drawing  functions. 
We  adapted  this  architecture  for  realtime  graphics 
with  a  double  buffering  concept  and  an  intelligent 
color  lookup  table  (CLUT),  which  also  support 
hardware  blinking  and  overlay  techniques.  The 
graphic  processor  is  coupled  via  a  dual  ported  RAM 
to  the  system  bus.  This  supports  an  asynchronous, 
highspeed  interface  with  fast  update  rates  (see  figure: 
8). 

33.  Video  and  image  conversion  modules 


Because  the  external  video  sensors  (  FLIR,  LLLTV 
etc.)  have  often  line  standards,  which  are  different 
from  those  of  the  displays,  a  conversion  to  the  display 
line  standard  is  required.  Basically  we  see  two 
different  kinds  of  conversion. 

3.5.1.  Video  conversion  modules 

This  is  a  simple  solution  (see  figure:  9)  for 
linestandard  conversion.  For  example  in  the  AVT 
program  we  convert  the  4:3  CCIR  interlace  video 
signals  to  the  1:1  50  Hz.  none  interlace  standards  of 
the  displays.  The  interlace  signal  is  interpolated  in 
realtime  with  just  one  line  duration  delay.  We  support 
thereby  realtime  scrolling  of  a  1:1  window  in  the  4:3 
area. 

33.2.  Image  conversion  modules 

If  the  conversion  (see  figure:  10)  requires  more 
intelligence,  this  has  to  be  done  by  putting  the 
incoming  video  into  a  framebuffer,  transforming  that 
information  to  another  framebuffer  and  reading  this 
second  framebuffer  at  the  outcoming  video  line 
standard.  This  transformation  has  to  support  functions 
like  translation,  scaling,  panning,  rotation,  zooming, 
scrolling  and  so  on.  This  works  with  a  minimum 
delay  of  one  frame. 

3.6.  Video  Mixing 

The  video-mixing  module  mixes  video  signals  of  the 
same  line  standard  in  an  analog  way.  Priority  and 
additive  mixing  are  supported. 

3.7.  Displays 

3.7.1.  HDD 

In  applications  for  realtime  graphics  there  are  some 
problems  with  the  existing  displays  on  the  market. 
Shadowmask  displays  have  problems  with  vibration 
due  to  the  mechanical  construction  of  the 
shadowmask  and  are  limited  in  brightness.  The 
existing  LCD  displays  are  slower  in  image  changes 
due  to  the  physical  switching  time  of  the  transistors. 

Because  of  these  reasons  we  propose  the  beam  index 
technology  (see  figure:  11).  In  displays,  using  this 
technology,  the  phosphor  is  arranged  in  vertical 
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stripes,  in  a  sequence  of  RGB  on  the  screen, 
separated  by  small  black  stripes,  for  contrast 
enhancement  There  is  no  shadow  mask  which 
absorbs  80%  of  the  beam  energy,  so  the  whole  energy 
is  concentrated  to  the  phosphor  stripes,  which  means 
much  more  brightness  on  the  screen.  The  resolution  is 
$12  by  $12  triple  (RGB)  on  a  S  by  S  inch  display. 
Because  there  is  no  shadow  mask,  this  display  has  no 
problems  with  vibration. 

3.7.2.  HMD 

On  the  X31A  programme  we  developed  a  binocular 
HMD  with  a  total  field-of-view  of  100  by  40  degrees 
with  a  high  resolution  of  2000  by  800  pixel 
(monochrome).  The  distortion  of  the  optical  system  is 
corrected  with  an  digital  deflection  correction  within 
the  displays.  This  ensures  a  linear  screen  area  with  a 
constant  angle/pixel  format.  The  1  inch  monochrome 
tubes,  which  feed  the  optical  part,  produce  high 
brightness  and  resolution. 


4.  SYSTEM  SOFTWARE 

The  basic  system  software  is  designed  fully  in  Ada. 
This  gives  the  application  programmer  a  high-level, 
abstract,  easy-to-use  interface  to  all  system  functions. 
Because  the  highlevel  Ada  interrupts  require  a  task- 
entry  with  context  switching  on  every  interrupt,  low 
level  interrupts  are  used  within  the  drivers.  This 
brings  short  interrupt  latency  and  maximum 
performance. 

In  addition  to  the  normal  device  handling,  the  graphic 
processing  parts  require  a  more  abstract  interface.  We 
provide  here  an  interface,  where  the  user  defines  a 
virtual  screen  for  every  image  fonnat  (see  figure:  12). 
In  that  virtual  screen  the  user  define  the 
transformation  from  the  aircraft  data  to  the  graphical 
objects  in  Ada.  This  virtual  screen  can  be  assigned  at 
runtime  very  easy  to  a  physical  display,  by  a  simple 
Ada  command.  If  the  same  screen  format  is  to  be 
shown  on  different  displays,  the  transformation 
algorithms  are  shared  and  the  display-file  is  just 
copied  to  more  graphic  processing  units,  which 
increases  the  overall  performance  (see  figure:  13). 

This  graphic  development  can  be  done  on  a 
workstation  prior  to  the  availability  of  the  final  target 


system.  Several  displays  are  emulated  in  a  X-window 
environment  with  exactly  the  same  Ada  interface  and 
graphic  functions  as  for  that  system.  So  the  graphic 
development  can  be  done  on  a  workstation  network 
with  many  workplaces,  before  going  to  the  final 
hardware  (see  figure  14). 

5.  SYSTEM  PERFORMANCE 

The  Ada  compilers,  which  are  available  today,  have 
the  same  performance  as  C  compilers  due  to  the 
sequential  part  of  the  language.  The  performance  per 
CPU  has  just  increased  from  3.5(68020/30)  to  20 
MIPS(68040).  This  means  that  the  PAH2  BCSG 
benchmarks  are  now  faster  by  factor  of  7  then  those 
on  the  68020/30  CPU. 

Besides  the  synchronous  rendezvous  mechanisms  in 
Ada,  an  asynchronous  task-to-task  communication  via 
mailboxes  with  semaphores  is  required.  This  function 
is  not  yet  defined  in  the  language  and  the  realization 
of  this  in  Ada  is  much  too  slow.  So  an  realtime 
operating  system  has  to  supply  this.  The  same 
problem  arises  with  multiprocessor  communication. 
The  realtime  mother  clock  quantization  in  the  Ada 
operating  system  is  now  decreased  from  20  msec 
(68030)  to  less  than  2  msec  (68040).  This  results  in  a 
finer  resolution  in  task  scheduling  and 
synchronization  (see  figure:  15). 

6.  DEVELOPEMENTENVIRONMENT 

First  we  started  with  the  Telesoft  Ada  Compiler 
combined  with  the  Ready-Systems  Operating  system 
ARTX.  Because  this  solution  is  no  longer  supported, 
we  have  now  switched  over  to  the  Telesoft-Ada- 
Compiler  combined  with  the  PSOS-M  operating 
system.  In  addition  to  ARTX,  PSOS-M  supports  the 
same  task  to  task  comnr.'nication  interface  for  single 
and  multiprocessing  and  is  an  object  oriented  kernel. 
This  makes  it  easy  for  the  application  programmer  to 
shift  tasks  from  one  processor  to  another  without 
changing  the  communication  to  other  processes. 

The  cross  and  host  development  tools  are  available  on 
UNIX  workstations  (see  figure:  16).  We  use  cadres 
teamwork  tools  SA/RT  for  the  system  analysis  phase 
to  build  the  system  requirements  model.  From  this  we 
construct  the  system  architecture  model  which 
represents  one  possible  architecture  to  fulfil  the 


requirements.  In  the  design  phase  we  work  with  cadre 
teamwork  Ada,  which  supports  the  Ada  structure 
graphs  (ASG),  defined  by  Buhr,  to  construct  the 
software  design.  From  these  Ada  structure  graphs  we 
generate  the  Ada  Source  Frame  Code.  This  Ada 
Frame  Code  is  finally  filled  with  the  appropriate  Ada 
statements  and  then  transfered  to  the  Ada-Compiler. 

7.  APPLICATION  EXAMPLES 

7.1.  AVT 

The  AVT  program  is  an  experimental  helicopter 
programme,  unsing  a  modified  BK117,  on  which 
flexible,  generic  avionics  have  to  be  installed.  The 
main  goal  is  to  build  generic  instances  of  this 
avionics  to  support  specified  missions.  MBB  will  do 
the  experimental  flights,  ESG  is  developing  the 
operational  Ada  flight  program  and  we,  the 
Telefiinken  System  Technik  have  developed  the 
hardware  and  basic  system  software  for  the  image  and 
mission  computing  system.  This  system  contains 
several  interfaces:  serial,  parallel,  ARINC-429,  MEL- 
1553,  three  intelligent  graphic  processing  units  with 
video  conversion  and  mixing,  thriven  by  three  CPUs. 

7 2  X31A-HMTAS 

This  system,  the  Helmet-Mounted-Target-Acquisition- 
System,  is  designed  for  flight  simulation.  It  contains 
a  binocular  HMD  (100  by  40  degree),  a  line  of  sight 
locator  (LOSL)  and  an  Image  Management  System. 
This  system  receives  the  position  of  the  simulated 
aircraft,  the  position  of  a  target  aircraft,  the  line  of 
sight  of  the  pilot  and  transforms  this  information  in 
order  to  display  a  target  wireframe  image  with  100 
3D  points,  connected  via  300  3D-vectors,  in  realtime 
(60  Hz.)  update  rate  to  the  HMD. 

73.  X31A-HDD 

One  aspect  of  the  X31A  programme  is  to  produce  a 
new  3D  flight  path  display  with  a  3D  globe-like 
object  (defined  by  MBB)  to  support  the  pilot  in  the 
post  stall  (low  speed,  high  pitch)  range  of  the  X31A 
aircraft  This  is  done  by  our  Image  Management 
System  which  transforms  the  aircraft  data  in  realtime 
into  a  3D  graphical  object  on  a  special  5  by  5  inch 
Beam  Index  Color-Display. 
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MICS  (Mission  and  Image  Computing  System):  bus  structures 
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MICS  (Mission  and  image  Computing  System): 
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Abstract 

The  concept  of  human-electronic 
co-operation  in  the  cockpit  is  synonymous 
with  that  of  a  team.  Whether  or  not 
team  members  interact  effectively  will 
rely  largely  upon  the  pilot's  acceptance 
of  his  electronic  team  mate.  This  paper 
reports  on  the  attitudes  of  eight  British 
Aerospace  test  pilots  towards  the  future 
of  such  co-operation.  Particular  emphasis 
is  laid  upon  the  factors  of  system 
function,  task  allocation  and  trust. 

Pilots  opinions  are  examined  against  a 
schema  of  'Operational  Relationships’, 
recently  proposed  in  the  literature. 

1  INTRODUCTION 

The  purpose  of  this  paper  is  to  address 
the  issues  of  trust  and  acceptance  in 
cockpit  human/electronic  teamwork.  There 
is  a  legitimate  concern  that  the  strategy 
of  automating  all  of  the  pilot  tasks 
which  it  is  technically  feasible  to 
automate  is  unlikely  to  provide  the 
optimum  design  for  the  future  human  - 
electronic  aircrew  team  (eg.  Hollister 
1986).  A  first  defence  against  this  can 
be  achieved  by  developing  a  close  liaison 
between  the  system  designer  and  the 
pilot  population.  This  should  help  in 
the  identification  of  those  tasks  whose 
automation  would,  in  the  opinion  of 
aircrew,  be  most  beneficial  and  thus 
enhance  the  likelihood  of  pilot  acceptance. 

British  Aerospace  (Military  Aircraft) 
Limited  employ  a  number  of  pilots  to 
perform  test  flying  on  aircraft  such  as 
the  Harrier,  Hawk  and  Tornado  Aircraft. 
These  pilots  have  many  years  of  fast  jet 
experience  in  the  Royal  Air  Force  or 
Royal  Navy,  Fleet  Air  Arm  as  well  as  in 
other  NATO  forces.  This  pool  of  experience 
offers  BAe  an  opportunity  to  gather 
opinions  and  gauge  initial  reactions  to 
the  specific  and  general  acceptability 
of  automation. 

Questionnaires  and  structured  interviews 
were  used  to  elicit  the  views  of  eight 
BAe  test  pilots  regarding  the  functions 
and  philosophies  that  should  drive  the 
integration  of  automated,  semi-automated 
and  human-electron!  co-operative 
technologies  in  the  cockpit.  The  pilots 
had  a  total  of  31400  hours  of  fast  jet 
flying  experience  (Details  are  given  in 
Section  1.1).  During  these  interviews 
reference  was  made  to  the  concept  of 
Operational  Relationships  (OR)  as 
described  by  Krobusek,  Boys  and  Palko 
(1988).  In  this  schema  ten  distinct 
categories  of  Operational  Relationship 
are  defined.  These  range  from  OR'A',  where 
the  pilot  performs  the  activity,  to 
OR’G’3,  where  the  system  may  perform  the 
action  autonomously.  All  10  are  listed 


below  in  Table  1,  and  were  used  after  the 
interviews  to  categorise  responses.  During 
the  interviews  the  concepts  were  used  to 
prompt  the  pilots  to  consider  the 
possibilities  and  potential  for  cockpit 
automation.  Throughout  this  paper 
opinions  are  related  to  this  schema. 

TABLE  1  'OPERATIONAL  RELATIONSHIP’  SUMMARY 
TABLE 

OR'A'  -  The  pilot  performs  the  activity 
OR'B'  -  The  (relatively  straightforward) 
activity  is  performed  automatically  by  the 
system . 

OR'C'  -  The  system  may  remind  the  pilot 
if  the  pilot  asks  or  has  authorised  such. 
OR'D'  -  The  system  may  remind  the  pilot 
OR*E'  -  The  system  may  prompt  the  pilot 
(with  unrequested  information). 

OR'F'  -  The  system  has  been  given  authority 
to  perform  function,  but  with  pilot 
consent . 

OR’G'  -  The  system  may  perform  an  action 
only  if  various  conditions  are  met. 

OR'G'l  -  The  system  may  perform  the  action 
but  must  concurrently  notify  the  pilot. 

OR'G'2  -  The  system  may  perform  the  action, 
but  must  notify  the  pilot  when  first 
convenient  for  the  pilot. 

OR'G'3  -  The  system  may  autonomously  perform 
the  action. 

The  interview  techniques  required  pilots  to 
iteratively  address  specific  elements  and 
aspects  of  the  piloting  task,  the  aircraft's 
systems  and  it's  operational  role.  This 
provided  a  flexible  structure  within  which 
pilots  could  consider  existing  automation 
requirements  as  well  as  future  possibilities. 
Four  general  areas  were  addressed,  these 
were  (i)  the  management  of  the  aircraft 
systems,  (ii)  situation  assessment,  (iii) 
tactics  and  (iv)  the  man  machine  interface. 

1.1  PILOT  EXPERIENCE 

Details  of  the  fast  jet  flying  experience 
of  the  eight  pilots  interviewed  are  given  in 
Table  2  below. 

Table  2  Approximate  Pilot  Logs 
PILOT  'iOURS 


1 

2 

3 

4 

5 

6 

7 

8 


4700 

3500 

3000 

3500 

4000 

3500 

4200 

5000 

Total  31400 
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Aircraft  flown  include: 

Tornado  (IDS/ADV) 

Jaguar 

Hunter 

Jet  Provost 

Hawk 

EAP 

Harrier  (+Sea) 

F16 

Phantom 

2.  MANAGEMENT  OF  AIRCRAFT  SYSTEMS 

2.1  Engines 

In  general  the  pilots  welcomed  engine 
automation,  although  they  could  not  easily 
envisage  the  potential  for  automation 
beyond  the  fully  digital  engine  controls 
current  in  GR5  Harrier  or  proposed  for  EFA. 
Nonetheless,  further  automation  would  be 
welcomed  if  it  maximised  opportunities 
for  the  pilot  to  assimilate  higher  level 
information  by  reducing  engine  system 
distractors.  Particular  emphasis  was  laid 
upon  the  desirability  of  the  system 
performing  all  pre-flight  checks  and  start¬ 
up  procedures  as  there  is  considerable 
pressure  upon  the  pilot  at  this  stage  in 
a  mission,  especially  if  'scrambled'.  At 
this  stage  in  a  mission  pilots  wished  to 
only  be  alerted  only  if  a  significant* 
system  failure  was  detected  (OR'G'l) 
believing  that  sel f-cor rec t ing  systems 
should  self-correct  autonomously  (OR'B'). 
Following  a  system  failure  it  was 
suggested  that  details  relating  to  the 
performance  penalty  of  that  failure 
would  be  required  as  the  pilot  may 
decide  to  fly  in  spite  of  the  failure. 

Thus  pilots  welcomed  decision  aiding  but 
did  not  wish  the  system  to  take  a 
FLY/NO-FLY  decision. 

In  general  pilots  believed  that  the  aim 
of  engine  automation  should  be  to  provide 
'care-free'  handling  particularly  during 
peiiods  of  high  mental  workload  as 
experienced  in  low  level  flight  and 
during  emergencies  or  combat  situations. 
This  could  only  be  achieved  if  the 
system  was  of  a  sufficiently  high 
integrity  to  engender  a  high  level  of 
trust. 

2.2  Fuel  and  Hydraulic  Systems 

Current  fuel  sysetm  automation  is 
considered  to  be  at  a  fairly  high  level, 
although  past  experience  has  shown  the 
importance  of  a  'transparent'  system 
that  enables  the  pilot  to  confidently 
assume  control  of  the  system  in  the 
event  of  a  failure.  Pilot  opinion  was 
entirely  in  favour  of  further  hydraulic 
automation,  although  there  was 
disagreement  concerning  the  OR  that 
should  govern  these  procedures  (ranging 
over  OR ' F ' ; OR ' G '  &  OR'G'l).  Pilots  stated 
that  they  would  w'sh  to  sanction  ( 0 R ' F ' ) 
any  procedure  that  would  aff’ct  aircraft 
performance  (eg.  moving  fuel  may  affect 
centre  of  gravity). 


1  'Significant'  in  this  context  refers  to 
factors  that  will  affect  flight  performance 
or  operational  capability.  


It  was  suggested  that  pilots  should  not 
have  to  bother  themselves  with  fuel  or 
hydraulic  system  operations,  although 
high  level  information  was  essenti; 1 
(eg.  range,  kg.  left,  undercarriage  status) 

It  was  recognised  that  the  requirement  for 
information  and  sanctioning  may  be  part  of 
the  process  of  developing  trust  in  an 
automated  system. 

2.3  Battle  Damage,  Faults,  Malfunctions 

In  general,  pilots  did  not  want  to  be 
informed  of  the  technical  diagnosis  of 
specific  types  of  battle  damage,  fault  or 
general  malfunction.  Rather,  under  these 
conditions  they  wanted  the  system  to 
reconfigure  following  OR'G'3  with  the 
qualification  that  should  operational 
capabilities  or  flight  performance  be 
affected  the  system  should  immediately 
inform  the  pilot  of  these  new  parameters. 
Again  this  is  an  example  of  the  need  for 
decision  aiding  requirements  to  parallel 
those  of  automation. 

2.4  Avionics 

Automation  of  navigation  systems,  which 
involve  many  routine  tasks,  was  believed 
to  be  a  sensible  goal  although  cautionary 
reference  was  made  to  the  integration  of 
early  automated  Inertial  Navigation  systems 
which  were  found  to  increase  vather  than 
reduce  workload  due  to  their  poor  reliability 
Although  opinion  differed  upon  the  level  of 
autonomy  (LOA),  see  also  Krobusek,  Boys  and 
Palko  (op  cit.)  at  which  specific  navigation 
and  other  avionic  systems  should  be  set, 
there  was  general  agreement  that  automated 
system  functions  should  remain  hidden  until 
a  pre-defined  point  at  which  the  system 
would  request  authorisation  to  continue, 
thus  in  effect  proposing  a  variant  of  CR'F'. 
The  point  was  reiterated  that  if  a  function 
could  be  automated  with  high  reliability 
and  the  effect  of  this  automation  had  no 
effect  upon  the  aircraft's  performance 
then  the  fact  of  that  automation  should 
remain  hidden  from  the  pilot.  However  as 
recommended  by  Krobusek,  Boys  and  Palko 
(op  cit.)  it  was  agreed  that  such  events 
be  recorded  for  later,  in-flight  perusal. 

This  appears  to  support  a  special  case 
of  OR  'B',  but  with  the  qualification  that 
such  events  be  recorded  in  case  they  impact 
upon  ether  factors  later  in  a  mission. 

3  SITUATION  ASSESSMENT 

3.1  Automated  Sensor  Management 

All  pilots  agreed  that  an  automated  sensor 
manager  that  presented  an  accurate  tactical 
'picture'  was  required,  hut  were  sceptical 
about  how  accurate  such  a  system  could  be 
due  to  the  number  of  variables  that  must  be 
considered  and  the  often  stated  requirement 
to  retain  flexibility.  Although  such 
flexibility  may  be  achieved  by  pre-sett  ing 
the  'goals'  of  a  sensor  manager's  LOA 
(eg.  be  stealthy  until  x  etc)  pilots  were 
in  general  unwilling  to  accept  the  concept 
of  L'sOA  at  a  more  complex  level  than  that 
of  sophisticated  tactical  decision  aider 
or  mission  management  aid.  Overall  the 
concept  of  L'sOA  as  interpreted  by  pilots 
at  the  highest  level  of  authority  did  not 
extend  to  that  of  dynamic  re-allocation  of 
f unc  t  ion  . 
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The  highest  acceptable  level  was  perceived 
to  be  that  of  pre-flight  presets  of  the 
functions  that  would  be  performed  by  the 
pilot  and  by  the  system.  The  consensus 
appeared  to  be  that  L'sOA  would  not 
(and  should  not)  be  reconf igurable  in 
flight.  This  view  appeared  to  be  driven 
by  the  realisation  that  dynamic 
re-allocation  of  function  and  in-flight 
LOA  resets  would  occur  during  high 
workload  periods  and  potentially  contribute 
to  confusion  during  highly  inopportune 
phases  of  a  mission.  It  seems  likely  that 
the  most  useful  arrangement  of  pilot-system 
co-operation  (LOA)  will  be  predictable 
because  it  will  be  necessary  to  reduce 
the  occurrence  of  variations  in  this 
relationship  during  periods  when  the 
pilot  is  integrating  the  tactical 
significance  of  many  external  variables. 

Pilots  did  agree  that  a  high  integrity 
automated  sensor  suite  would  be  extremely 
useful  and  afford  a  significant  combat 
advantage  for  the  pilot.  Recent 
developments  such  as  auto-scan  centering 
and  auto-scan  volume,  as  used  in  radar 
target  acquisition,  have  been 
enthusiastically  received  due  to  the 
accompanying  large  reductions  in 
workload.  Although  pilots  believed 
that  automated  sensor  management  and 
sensor  correlation  were  priorities,  they 
were  concerned  about  the  integrity 
of  such  a  system.  Pilots  suggested  that 
trust  and  confidence  in  such  a  system 
could  only  be  brought  about  through 
repeated  trials  in  which  the  auto-sensor's 
'picture'  was  found  to  be  more  accurate 
than  that  which  the  pilot  had  developed 
from  the  usual  sensor  sources.  It  was 
suggested  that  it  would  be  essential  to 
attach  confidence  levels  to  the  fused 
and  correlated  output  of  such  systems. 

Thus  sensed  information  could  be  presented 
in  a  form  such  as  "I'm  70%  sure  this  is 
a  Flanker".  Given  these  integrity  and 
probability  pre-conditions,  pilots 
believed  that  they  would  accept  sensor 
management,  correlation  and  fusion  at 
0R’G'3. 

3.2  Automated  Defensive  Systems  (DAS) 

Defensive  aids  systems  automation  was 
generally  considered  a  good  idea, 
although  pilots  were  concerned  that  the 
system  could  easily  be  spoofed  (tricked 
into  making  an  error  of  commission,  a 
false  identification  of  a  threat).  To 
cope  with  this  eventuality  most  pilots 
believed  that  an  OR ’ G ' 1  level  would  be 
required  but  also  mentioned  that  the 
need  to  regularly  monitor  the  system 
to  detect  spoofing  might  increase 
workload.  As  with  most  systems  manual 
override  was  considered  essential. 

All  pilots  were  unanimously  opposed  to  the 
concept  that  the  DAS  should  be  linked 
to  the  flight  control  system  (FCS)  such 
that  automated  missile  'Break'  procedures 
could  be  undertaken  without  forewarning. 
Although  several  rationales  were  provided, 
(including  those  of  system  error, 
spoof /annoyance  factors  and  the  potential 
for  physical  injury)  opposition  to  this 
proposal  was  sufficiently  strong  to 
suggest  that  automated  FCS  intervention 
'went  against  the  grain’  at  a  fundamental 


level.  Missile  'Break'  related  automation 
was  acceptable  only  at  OR'E'  (eg.  BREAK 
PORT). 

4  TACTICS 

There  was  little  agreement  concerning  the 
usefulness  of  automated  tactics  systems 
(ATS)  but  all  felt  that  tactics  would  be 
the  most  complex  pilot  tasks  to  automate 
due  to  the  inherent  dynamic  and  flexible 
nature  of  combat.  As  discussed  previously, 
pilots  appeared  reluctant  or  unwilling  to 
conceive  of  a  tactical  level  of  human- 
electronic  co-operation  that  exceeded  that 
of  sophisticated  decision  aid.  Interestingly 
the  point  was  made  that  a  capability  to 
vary  tactical  L'sOA  (on  the  ground)  may  be 
useful  as  a  pilot  training  aid  for  less 
experienced  pilots,  although  it  was  stated 
that  the  logic  and  reasoning  employed  by 
the  system  must  be  very  clear.  Pilots 
felt  that  the  optimum  role  for  ATS  would 
be  the  computation  of  target  engagement 
paths,  missile  release  zones  and  paths  of 
egress.  Most  pilots  believed  that  these 
functions  should  operate  at  OR'E'  levels, 
although  some  pilots  felt  that  they  may 
wish  to  allow  the  system  to  carry  out  the 
engagement  through  sanctioning  system 
control  of  the  FCS  (OR'F').  All  pilots 
agree  that  regardless  of  the  OR  covering 
target  engagement  the  pilot  must  perform 
the  weapons  release  task  himself  (OR'A1). 

One  pilot  could  see  the  full  potential  for 
this  type  of  automation  stating  that 
should  the  pilot  delegate  target  engagement 
procedures  to  the  automated  system,  this 
would  - 

"....allow  one  aircraft  to  almost  have 
the  capability  of  two,  as  the  pilot  will 
be  able  to  cover  against  threats  and  check 
systems  just  as  a  second  crew  member  would 
do.  ” 

There  was  a  general  feeling  amongst  the 
pilots  that  although  they  could  imagine 
the  potential  role  of  an  ATS  decision  aid 
they  would  have  difficulty  trusting  the 
validity  of  these  displays  or  indeed  the 
information  upon  which  they  were  based. 

A  typical  comment  concerned  the  auto¬ 
detection  of  a  SAM  site,  it  was  believed 
that  pilot's  would  wonder  (a)  is  it  really 
a  SAM  site?  (b)  has  the  site  run  out  of 
missiles?  (c)  is  it  just  illuminating 
(spoofing)  with  it’s  radar?  Pilots  felt 
on  the  whole  that  they  would  be  reluctant 
to  trust  such  a  system  or  the  data  upon 
which  it  made  its  decisions.  Two  general 
accompanying  comments  were  made  these 
were  that  : 

(1)  The  most  useful  tactical  decision 
aiding  would  be  the  identification  of 
targets  (Automated  sensor  management) 
together  with  details  of  target  performance 
capabilities  together  with  own  optimum 
engagement  parameters  (eg.  intercept 

speed  and  course). 

(2)  The  tactical  automation  'nightmare'  is 
that  the  automated  aircraft  provides  lots 
of  clear  tactical  information  to  the  pilot, 
but  that  this  information  is  wrong  because 
the  system  is  being  spoofed. 
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5  NAN  MACHINE  INTERFACE  (MMI) 

All  pilots  agreed  that  the  MMI  of  automated 
systems  would  be  critical  to  aircrew 
acceptance  of  such  systems.  A  major 
concern  was  that  the  pilot  should  be 
presented  with  an  unambiguous  display  of 
'who'  was  in  control  of  'what'.  There  was 
also  concern  that  the  pilot's  desire  to  be 
told  what  the  system  was  doing  (OR'G'l) 
or  to  sanction  automated  actions  (OR'F') 
might  actually  increase  his  workload. 

It  was  unanimously  agreed  that  there  is 
already  too  much  displayed  information  in 
the  cockpit  for  the  pilot  to  reliably 
intake  at  periods  of  high  workload  and 
that  the  proliferation  of  sensor  and 
weapon  aiming  systems  will  exacerbate 
this  problem,  particularly  in  single  seat 
aircraft.  Consequently  an  MMI  priority 
is  to  reduce  the  amount  of  information 
presented  in  the  cockpit  by  concentrating 
on  the  fused  and  correlated  'high  level’ 
information  to  be  presented  to  the  pilot 
(eg.  "port  flap  hydraulic  failure"  or 
"port  flap  stuck  at  15  degrees"  is  less 
useful  than  starboard  roll  reduced  to 
**"etc).  Thus  the  pilot  should  immediately 
be  aware  only  of  the  fact  and  the 
implications  of  significant  changes  within 
or  outside  his  aircraft.  The  significance 
of  an  event  will  require  clarification 
through  further  research,  nonetheless 
significance  appears  related  to  the  impact 
of  events  upon  operational  performance 
tactics  and  safety. 

Most  pilots  agreed  that  during  periods  of 
high  workload  it  would  be  extremely 
advantageous  if  an  automated  system  could 
prioritise  information  and  present  this  at 
a  time  when  this  would  not  be  distracting. 
This  is  similar  to  a  special  case  of 
OR'G'2  in  which  the  concept  of  'performing 
an  action'  is  changed  to  'gathering 
information',  rendering  the  nature  of  the 
human-electronic  interaction  closer  to 
that  of  human  co-operation  rather  than  a 
simple  shift  of  the  locus  of  executive 
control  . 

6  GENERAL  ISSUES 

A  number  of  general  issues  emerged  from 
the  interviews  that  have  bearing  upon 
the  integration  of  the  human-electronic 
team  and  the  pilots  ability  to  speculate 
upon  such  a  relationship. 

6.1  Operational  Relationships 

The  concept  of  OR's  was  easily  understood 
by  all  pilots,  although  the  pilots  varied 
in  the  level  of  the  OR  they  were  prepared 
to  allocate  to  human-electronic  teamwork. 
This  in  itself  may  support  the  concept 
of  'Pilot  Tailoring'  a  process  which 
would  essentially  customise  a  pilot's 
individual  LOA  requirements.  It  was 
suggested  that  the  ten  OR's  proposed  by 
Krobusek,  Boys  and  Palko  (op  cit.)  could 
in  fact  be  simplified  for  the  purposes 
of  gauging  pilot  opinion  to: 

a)  The  system  does  it  always  (0R'B','G'3) 

b)  The  system  does  it  sometimes  ( 0 R ’ G '  ) 

c)  The  system  does  it  and  tell  the  pilot 

(either  then  or  later)  (OR'G'l, ' G ’2) 

d)  The  system  asks  the  pilot  to  be  allowed 
to  do  it  (OR'F') 


e)  The  pilot  does  it  (OR  A) 

Those  OR's  omitted  from  the  original  schema 
(OR  'C','D’,’E')  appear  qualitatively 
different  from  the  rest  and  as  such  may  be 
better  suited  to  a  schema  describing  levels 
of  decision  aiding. 

6.2  Levels  of  Autonomy 

In  general,  pilots  had  some  difficulty 
imagining  functional  models  of  LOA  concepts. 
There  existed  a  general  resistance  to  the 
concept  that  human-electronic  team 
co-operation  could  be  redefined  whilst 
in-flight.  Although  the  concept  of 
'pilot  tailoring'  was  welcomed  it  was 
believed  unlikely  that  these  parameters 
would  be  re-tailored'  between  missions  due 
to  the  sheer  complexity  of  remembering 
another  set  of  variables.  A  point  made 
throughout  all  the  interviews  was  that 
reducing  the  complexity  of  aircraft 
systems  must  be  the  goal  of  automation. 
Pilots  added  that  they  may  well  not 
interact  with  systems  that  added 
significant  complexity  to  their  task  even 
if  those  systems  could  buy  an  operational 
advantage.  Although  Krobusek,  Boys  and 
Palko  (op  cit.)  argue  that  the  end  product 
of  integrating  an  LOA  approach  within 
automated  aircraft  systems  would  buy  a 
"very  dynamic  range  of  performance"  for  the 
system,  pilots  appear  more  concerned 
that  they  should  understand  exactly  what 
the  performance  characteristics  of  all 
their  aircraft  systems  will  be  throughout 
an  entire  mission,  an  assumption  that  does 
not  allow  for  a  wide  range  of  in-flight 
variations  to  the  co-operative 
human-electronic  team  relationship.  The 
LOA  concept  did  receive  support  from  some 
pilots  who  suggested  that  it  would  provide 
a  useful  training  and  combat  aid  for  the 
inexperienced  pilot. 

7  CONCLUSION 

Overall,  the  pilots  welcomed  automation 
that  would  relieve  them  of  tasks  during 
periods  of  high  critical  workload  and  of 
carrying  out  mundane  and  routine  monitoring 
tasks.  Whilst  there  is  a  degree  of  mistrust 
and  scepticism  concerning  the  integrity  and 
reliability  of  future  automated  systems, 
the  development  of  such  systems  are 
enthusiastically  supported  as  they  are 
seen  as  the  only  means  by  which  the  pilot 
will  be  able  to  cope  with  the  workload 
demands  anticipated  from  forthcoming 
aircraft  systems.  However,  it  appears  that 
and  effect  of  this  underlying  mistrust  is 
that  most  of  the  pilots  interviewed  wish 
to  be  presented  with  information  on  at 
least  some  aspects  cf  the  automated 
decision  making  processes,  a  requirement 
which  might  actually  increase  the  workload 
associated  with  a  given  task.  Interestingly, 
the  pilots  opinions  were  similar  to  those 
in  the  sample  reported  by  Taylor  (1988) 
in  venturing  that  trust  in  automated  systems 
would  not  actually  develop  through  the 
presentation  of  premises  and  hypotheses 
upon  which  automated  decisions  had  been  made 
but  that  an  individual’s  trust  would  develop 
when  the  system  repeatedly  'got  it  more 
right'  than  the  pilot.  Ultimate  acceptance 
of  highly  automated  systems  would  be 
achieved  only  when  the  'folklore'  of 
trustworthiness  generated  by  reliable 
systems  is  passed  onto  the  next  generation 
ai  ••  •  !  — - - 
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Many  pilots  expressed  a  strong  concern 
that  automation  will  be  introduced 
without  fully  taking  into  account 
the  tasks  that  the  pilot  performs 
resulting  in  a  system  that  will  not  be 
used  or  liked. 

The  sample  of  pilots  interviewed  in  this 
survey  was  relatively  small  and  hence 
their  opinions  should  not  be  considered 
representative  of  the  pilot  population 
as  a  whole.  Their  experience  and 
backgrounds  may  have  tended  to  encourage 
a  greater  caution  and  apprehension  of 
automation  concepts  than  would  be  found 
amongst  those  pilots  who  are  currently 
joining  squadrons. 

Finally,  it  should  be  recognised  that 
pilot  opinions  are  just  that,  they  may 
be  wrong,  they  undoubtedly  differ  and 
they  will  probably  change.  However, 
utimately  pilot  opinion  will  determine 
whether  or  not  the  human-electronic 
team  members  really  do  work  together 
as  a  team. 
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SUMMARY 

An  important  aspect  of  a  pilots  situational  awareness  is  the  need 
for  accurate  real  time  information  on  the  operational  status  of  all 
aircraft  systems.  False  and  intermittent  indications  have  been  a 
problem  with  many  of  the  built-in-test  (BIT)  functions  in  aircraft 
systems.  These  indications  result  in  Retest  OK  (RTOK)  and 
Cannot  Duplicate  (CND)  maintenance  events  when  the  aircraft 
returns.  These  types  of  events  account  for  35%  to  65%  of  the 
indicated  faults  in  many  Air  Force  avionics  systems.  Any  false 
indications  put  an  unnecessary  and  potentially  fatal  burden  on  the 
pilot  during  the  operational  scenario  and  also  consume 
significant  maintenance  resources. 

Many  of  these  false  alarms  and  intermittent  status  indications  are 
related  to  the  environmental  conditions  present  at  the  time  of  the 
indication.  Time  Stress  Measurement  Device  (TSMD)  technology 
offers  a  means  of  providing  this  crucial  environmental 
information  to  the  system's  BIT.  TSMDs  are  digital 
environmental  measurement  and  recording  devices  in  a 
microelectronic  package  which  can  be  embedded  into  a  system  at 
the  time  of  manufacture  or  on  a  retrofit  basis.  Hie  information 
collected  and  provided  by  the  TSMD  can  be  provided  in  real  time 
for  the  on-board  BIT  to  try  and  discriminate  between  transient 
system  performance  anomalies  and  hard  failures.  Thus,  only 
accurate  performance  status  information  is  reported  to  the  pilot 
The  paper  describes  the  background  of  TSMD  development 
current  state-of-the-art  in  TSMD  hardware  and  software,  current 
applications  which  address  the  enhancement  of  on-board  BIT 
performance  and  future  thrusts  in  the  TSMD  area. 

BACKGROUND 

The  magnitude  and  duration  of  environmental  and  electrical 
stresses  play  an  important  role  in  the  useful  lifetime  and  failure  of 
electronics/avionics.  Both  the  magnitude  and  duration  of  the 
stresses  affect  the  performance  and  failures  of  the  equipment 
Some  of  the  major  stress  sources  which  have  been  either 
identified  or  postulated  as  contributors  to  performance  changes 
and  failure  are  thermal  cycling,  thermal  soaks,  vibration,  shock, 
humidity,  corrosion  and  the  amount  of  energy  dissipated  in  the 
equipment  The  effect  of  these  stresses  produces  changes  and 
damage  in  the  parts,  interconnections  and  physical  structure  of 
equipment  which  leads  to  failure  or  altered  performance. 

Current  parameters  which  are  used  as  a  measure  of  reliability  such 
as  Mean-Time-Between-Failures  (MTBF)  are  measured  in  terms 
such  as  calendar  time,  operating  cycles,  etc.  Thus,  when  failures 
occur  we  are  capturing  only  the  effect  of  stresses  on  the  equipment 
and  not  the  actual  stress  history  which  the  equipment  was  exposed 
to  and  caused  it  to  fail.  The  TSMD  is  the  measurement  and 
recording  instrument  for  the  key  stress  parameters  of  the 
equipment  of  interest. 

A  TSMD  is  an  integrated  sensor  package  which  measures  and 
digitally  records  selected  environmental  or  electrical  conditions 
which  are  present  at  the  sensor  input.  The  stored  data  can  be 
subsequently  retrieved  for  use  and  analysis  in  the  maintenance  and 
diagnostic  process.  A  block  diagram  of  a  generalized  TSMD  is 


shown  in  Figure  1 .  The  data  flow  through  a  generalized  TSMD 
starts  with  the  analog  signal  output  of  the  various  sensors  being 
fed  into  any  necessary  signal  conditioning  circuitry  and  then  the 
signal  is  digitized  in  an  analog  to  digital  converter.  The  output  of 
the  A/D  converter  then  goes  to  the  controller  (microprocessor) 
for  any  necessary  manipulation  or  compression  and  then  is  stored 
in  nonvolatile  memory.  A  real  time  clock  is  used  to  provide  a 
timing  input  to  the  controller  and  a  data  port  to  the  controller  is 
used  as  a  user  interface  for  retrieval  of  stored  data  and  for 
programming  the  TSMD. 

DISCUSSION 

TSMD  Module  Development 

Beginning  in  Fiscal  Year  (FY)  86,  the  development  of  a  TSMD 
module  was  begun  by  Rome  Laboratory.  This  contractual 
program  with  Honeywell  was  jointly  funded  by  RL  and  the 
Productivity  Reliability,  Availability,  Maintainability  (PRAM) 
and  Generic  Integrated  Maintenance  and  Diagnostics  Systems 
(GIMADS)  Program  Offices  of  the  Air  Force's  Aeronautical 
Systems  Division  at  Wright-Patterson  AFB  OH.  A  TSMD  module 
using  off-the-shelf  components  was  designed,  developed  and 
qualified  for  use  in  a  flight  data  collection  program  utilizing  A-10 
and  A-7  aircraft.  The  module,  which  was  approximately  3  cm  x 
8  cm  x  16  cm,  measured  and  recorded  parameters  for  temperature, 
vibration/shock,  relative  humidity,  prime  power  voltage  and 
corrosion.  The  TSMD  module  was  cable  connected  to  a  battery 
pack  utilizing  sealed  lead  acid  cells  which  furnished  power  for 
continuous  TSMD  operation  and  recording.  The  data  from  the 
TSMD  module  was  removed  through  a  serial  port  which  was  used 
to  drive  a  RS-232  interface  through  a  small  adapter  box  into  a 
handheld  computer. 

Internally,  the  TSMD  circuitry  is  divided  into  a  sensor/analog 
circuit  card  assembly  (CCA)  and  a  digital  CCA.  The  TSMD  module 
was  designed  to  sample  the  environment  through  use  of  a  suite  of 
sensors  consisting  of  a  thermocouple,  humidity  sensor, 
corrodible  resistor,  a  single  axis  accelerometer  and  rectifier. 

Excursions  of  the  avionics  environment  beyond  user  defined 
limits  for  humidity,  vibration/shock,  and  aircraft  power  were 
recorded  in  memory  "bins”.  The  TSMD  measured  accumulated 
vibration  exposure  above  two  field  adjustable  thresholds  in  each 
of  four  different  frequency  bands  between  10  and  2000Hz.  The 
number  of  shock  events  greater  than  a  field  adjustable  threshold 
were  recorded.  The  accumulated  exposure  time  above  a  field 
adjustable  threshold  foT  relative  humidity  was  also  recorded.  The 
aircraft  prime  power  voltage  was  monitored  and  accumulated 
exposure  time  for  both  high  and  low  voltage  conditions  beyond 
predefined  limits  were  recorded. 

During  each  day,  any  excursion  of  a  set  parameter  beyond  its 
prescribed  limit  caused  the  corresponding  bin  to  be  incremented. 
At  the  end  of  the  day,  each  bin  contained  the  number  of  times  the 
limit  was  exceeded  for  the  day.  The  bin  counts  were  saved  and 
then  cleared  to  start  the  next  days  collection. 
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USER 

INTERFACE 


FIGURE  1 

TSMD  BLOCK  DIAGRAM 


Temperature  data  was  sampled  every  fifteen  seconds  and  recorded. 
Also,  a  log  of  power  on  and  off  events  is  included  in  the 
temperature  Hie.  In  this  way,  a  complete  record  of  the  temperature 
profile  seen  by  the  TSMD  was  recreated  during  data  analysis. 
However,  because  the  memory  space  required  to  store  a 
temperature  point  each  fifteen  seconds  was  prohibitive,  the  data 
was  compressed  to  reduce  the  data  storage  memory  size 
requirement.  The  compressed  data  takes  the  form  of  best  fitting 
ramps  and  lines  to  the  individual  data  points.  Resolution  of  the 
compressed  temperature  data  was  approximately  +/-  2.5°C. 

Sensing  of  corrosive  atmosphere  was  provided  by  a  detector 
consisting  of  two  mild  steel  strips.  One  strip  had  a  protective 
coating  and  served  as  a  reference  while  the  other  was  exposed  to 
the  atmosphere  and  exhibited  an  increase  in  resistance  to  indicate 
the  relative  amount  of  corrosion. 

The  realization  of  this  development  was  the  installation  of  ten  of 
the  TSMD  modules  in  five  A-10  aircraft  and  five  A-7  aircraft  (Ref 
1).  This  was  the  first  time  that  a  device  of  this  type  has  been  used 
to  collect  data  on  operational  aircraft  during  normal  sorties. 
Collection  of  data  of  this  type  prior  to  the  use  of  a  TSMD  usually 
required  much  larger  instrumentation  and  a  dedicated  aircraft  for 
testing.  A  typical  temperature  profile  for  a  day  including  two 
sorties  is  shown  in  Figure  2.  The  actual  flight  times  from  the 
squadrons  records  are  added  to  the  graph  to  provide  a  comparison 
with  the  power  on  time. 

Micro  TSMD 

A  natural  progression  from  the  module  was  to  an  integratec  unit 
the  size  of  a  microelectronic  package.  RL  began  development  of 
a  Micro  TSMD  in  FY88.  DTRI-Honeywell  and  Westinghouse 
competed  in  a  design  definition  phase  (Phase  I)  for  development 
of  a  hybrid  device  which  would  be  self-contained  (including 
sensors)  to  provide  temperature,  vibration,  shock,  and  voltage 
monitoring.  This  led  to  a  full-scale  development  phase  (Phase  D) 
of  the  Micro  TSMD.  The  Micro  TSMD  development  is  being 


sponsored  by  the  Air  Force  Reliability  and  Maintainability 
Technology  Insertion  Program  (RAMTIP).  DTRI-Honeywell  is 
under  contract  to  complete  the  design  of  the  Phase  II  Micro 
TSMD.  The  Micro  TSMD  is  suitable  for  mounting  on  a  circuit 
card  in  an  LRU.  The  first  insertion  of  the  Micro  TSMD  will  occur 
in  systems  supported  by  Wamer-Robins  Air  Logistics  Center 
(WR-ALC)  Robins  AFB  GA. 

The  Micro  TSMD  is  physically  a  1"  x  2"  flatpack  with  leads  on  50 
mil  centers.  Power  consumption  is  about  100  milliwatts  while 
powered  from  the  host  (recording  data).  When  host  power  is 
unavailable,  recording  activity  is  determined  by  the  availability 
of  a  button-cell  battery.  With  a  battery  present,  the  real-time 
clock  is  maintained  and  mechanical  shocks  are  recorded.  The 
recording  of  mechanical  shocks  is  useful  for  determining  if  units 
were  mishandled  diving  shipping  or  installation  of  the  host  card. 

Retrieving  stored  data  is  accomplished  by  an  RS-232  link  to  a 
debrief  computer  (laptop  model).  The  data  will  be  transferred  to 
disk  and  analysis  can  take  place  at  the  depot  or  other  maintenance 
level  in  the  logistics  process.  Programming  options  and 
parameters  can  be  changed  while  communicating  with  the  TSMD. 
The  TSMD  can  also  be  reset  by  the  debrief  computer. 

The  Micro  TSMD  is  a  hybrid  device.  It  contains  two  hybrid 
substrates,  a  digital  hybrid  and  an  analog  hybrid.  The  digital 
hybrid  contains  a  87C51  CPU,  EEPROM  memory,  a  real-time 
clock,  A/D  converter,  crystals  to  support  the  clocks,  and  glue 
logic. 

The  analog  hybrid  contains  the  temperature  sensor,  a 
piezoelectric  accelerometer,  the  transient  monitoring  circuit, 
differential  amplifiers,  voltage  regulation,  a  power  on/off 
detection  circuit,  and  voltage  monitoring  circuits.  The  vibration 
sensor  provides  a  transfer  function  which  will  reduce  aliasing  of 
the  sampled  signal.  Further  filtering  of  the  vibration  signal  is 
contained  within  the  analog  hybrid. 


COMPARTMENT  TEMPERATURE  (F) 
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CLOCK  TIME  (HOURS) 


FIGURE  2 

TSMD  STRESS  PROFILE  FOR  A-7 


The  sensors  and  given  digital  hardware  allow  recording  of 
parameters  with  the  range  and  resolution  shown  in  Figure  3. 
Software  programmable  options  allow  these  parameters  to  be 
recorded  in  many  ways. 

•  TEMPERATURE:  -55°C  to  125°C 
RANGE,  1°C  RESOLUTION 

•  VIBRATION:  +/-  3  Gs  MAXIMUM, 
20-2,000  Hz  PASSBAND,  -24dB, 
OCTAVE  ANTIALIASING  FILTERING 

•  SHOCK:  3.0  G  POSITIVE 
THRESHOLD,  -3.0  G  NEGATIVE 
THRESHOLD,  IMS  duration,  +  25  Gs 
MAXIMUM  SCALE 

•  DC  VOLTAGE:  0-10  V  RANGE,  10 
mV  RESOLUTION 

•  VOLTAGE  TRANSIENTS:  17  V  AND 
40  V  POSITIVE  THRESHOLDS,  4  V 
NEGATIVE  THRESHOLD,  1 
MICROSECOND  MINIMUM 
TRANSIENT  DURATION 


The  Micro  TSMD  software  is  designed  to  make  efficient  use  of  the 
available  memory  for  storage  of  recent  data  collected  and 
historical  data.  Also,  the  software  will  allow  the  user  to  change 
threshold  limits,  tolerance  regions,  sampling  rates,  memory 
allocation,  time  stamp  resolution,  and  maintenance  text  data. 
Examples  are  shown  in  Figure  4. 

•  TIME-LINE  DATA  ROLL  OVER 

•  TEMPERATURE  SAMPLING  PERIOD 

•  TEMPERATURE  AND  VOLTAGE 
RECORDING  TOLERANCE 

•  VOLTAGE  DEAD  BAND 

•  VIBRATION  SPECTRUM  SCALE 

•  INTERNAL  OR  EXTERNAL  SENSORS 

•  CLOCK  BATTERY  AND  SHOCK 
BATTERY 

•  TIME-LINE  DATA  FILE  ALLOCATION 


FIGURE  3 

MICRO  TSMD  SENSING  PERFORMANCE  SUMMARY 


FIGURE  4 

SOFTWARE  OPTIONS 


The  program  which  controls  the  Micro  TSMD  is  approximately 
36K  x  8  long.  Part  of  the  program,  16K  x  8  is  stored  in  the 
87C51  EPROM  with  the  remainder  stored  in  the  EEPROM. 
Scratch-pad  memory  built  into  the  87CS1  is  used  for  many 
purposes.  The  main  thrust  is  to  use  the  CMOS  scratch-pad  ram  for 
frequently  changing  data  (sampled  signals,  flags,  variables,  etc.) 
while  using  the  EEPROM  to  store  processed  data.  This  technique 
is  used  to  reduce  the  write  cycles  to  the  EEPROM  since  EEPROM 
has  a  wear  out  proportiona]  to  the  number  of  write  cycles. 

The  user  may  want  to  emphasize  one  environmental  parameter 
recording  more  than  others.  As  part  of  the  setup,  the  user  can 
define  how  memory  "files"  are  used  to  customize  the  Micro  TSMD 
for  stress  data  recording.  A  file  is  a  2K  x  8  segment  of  EEPROM 
memory.  The  main  types  of  flies  are  maintenance  text,  stress  data 
recording,  peak  event  record  and  post  record  analysis. 

Stress  data  recording  consists  of  temperature,  voltage,  shock  and 
voltage  transient  recording.  The  number  of  flies  allocated  for 
each  of  these  data  types  may  be  specified  by  the  user  depending 
on  his  requirements  and  limited  only  by  the  amount  of  memory  in 
the  Micro  TSMD. 

Peak  events  are  recorded  in  a  file  and  stored  with  a  time  stamp. 

The  number  of  peak  events  which  can  be  recorded  in  a  file  depends 
upon  the  resolution  of  the  time  stamp  specified  by  the  user. 

A  Fast  Fourier  Transform  is  performed  on  the  vibration  data 
immediately.  Records  of  time  at  overall  RMS  levels  and  time  at 
g^/Hz  levels  are  retained.  Temperature  stress  data  is  reduced  to 
total  time  on  temperature,  temperature  cycles,  and  temperature 
ramps  for  both  rise  and  fall. 

Information  in  a  stress  data  file  is  also  analyzed  to  gather 
information  from  it  that  would  be  useful  to  the  user.  Analysis  on 
that  data  to  reduce  it  to  a  final  form  will  free  up  memory  and  allow 
subsequent  stress  data  to  be  recorded  over  older  data  without 
losing  the  stress  history.  The  types  of  post  record  analysis 
include  temperature  and  voltage. 

Voltage  measurements  are  recorded  in  two  tables.  One  holds  total 
time  at  voltage  (5.0  volts  nominal).  A  second  table  holds  the 
number  of  on-time  durations. 

A  capability  exists  in  the  Micro  TSMD  to  accommodate  remote 
sensors.  This  flexibility  allows  the  user  to  add  enhanced  sensors 
as  they  become  available.  Additionally,  spare  analog-to-digital 
channels  are  brought  out  to  pins  which  allow  user  defined  signals 
to  be  processed  by  the  Micro  TSMD.  (Ref  2) 

Smart  BIT 

Concurrently  with  the  development  of  TSMD  technology,  Rome 
Laboratory  has  been  investigating  enhanced  techniques  for 
incorporation  into  a  system's  BIT  for  reducing  the  number  of  false 
removals.  These  investigations  have  been  carried  out  under  the 
title  "Smart  BIT".  Smart  BIT  is  best  thought  of  as  an  adjunct  to 
the  actual  functional  test,  but  it  could  easily  be  integrated  into  a 
singular  BIT  function.  Current  BIT  technology  often  places 
100%  confidence  on  the  results  of  a  test,  even  though  these 
results  could  be  biased  by  the  behavior  of  other  units  or 
temporarily  influenced  by  transient  environmental  conditions. 
Incorporation  of  an  N-out-of-M  filter  can  improve  the  condition 
to  some  degree,  yet  even  it  can  be  easily  misled.  Smart  BIT  goes 
beyond  these  simplistic  approaches  to  include  a  more  robust 
reasoning  process  that  looks  for  information  in  the  pattern  of 
faults  and  incorporates  knowledge  of  time,  the  environment,  and 
other  information  outside  of  the  functional  realm  of  BIT.  The 
principal  techniques  identified  for  which  software  was  developed 


and  demonstrated  were  Information  Enhanced  BIT,  Improved 
Decision  Rule  BIT,  Temporal  Monitoring  BIT  and  Adaptive  BIT. 

For  Information  Enhanced  BIT,  decisions  are  based  on 
information  internal  to  the  unit  under  test  (UUT)  as  well  as  other 
external  sources.  These  could  be  environmental  monitors  or 
information  concerning  the  operational  mode  of  the  platform  or 
the  health  of  other  systems.  Improved  Decision  Rule  BIT 
incorporates  a  structure  suggestive  of  an  expert  system  format  to 
increase  the  robustness  of  the  BIT  decision  process. 

Temporal  Monitoring  BIT  uses  Markov  modeling  techniques 
combined  with  a  finite  state  machine  representation  of  unit  health 
to  monitor  performance  over  time.  Adaptive  BIT  makes  use  of 
two  general  learning  paradigms:  k-nearest  neighbor  and  neural 
network  back  propagation.  In  both  cases,  the  BIT  report  in 
question  is  plotted  into  an  n-space  defined  by  the  various 
parameters  of  interest,  such  as  vibration,  GO/NO-GO,  airspeed, 
duration  of  failure,  etc. 

An  important  item  to  note  is  that  no  Smart  BIT  technique  may 
suffice  for  a  given  equipment  selection.  Careful  attention  must  be 
made  regarding  the  proper  selection,  prioritization  and 
integration  of  Smart  BIT  techniques  based  on  a  sound 
understanding  of  the  BIT  and  mission  needs  of  the  unit-under-test 
to  which  they  are  being  applied  and  to  the  specific  tailoring  of 
the  techniques  to  that  application. 

Research  has  been  done  to  define  the  degree  to  which  TSMD  and 
Smart  BIT  need  to  share  information  and  to  identify  pertinent 
characteristics  of  that  data  to  be  retained  in  memory.  It  is 
apparent  that  TSMD  data  should  be  available  at  three  levels  of 
temporal  resolution:  uncompressed  in  the  temporal  vicinity  of  a 
possible  failure  for  use  by  onboard  BIT,  compressed  for  duration 
of  a  mission,  and  statistically  characterized  for  equipment  and 
missions  to  be  used  in  long-term  trend  analysis. 

A  scenario  for  the  operation  and  use  of  TSMD  and  Smart  BIT 
technology  can  be  postulated.  During  a  mission  the  TSMD 
portion  is  continually  recording  stress  profiles  in  a  wraparound 
fashion,  replacing  old  data  with  ore  recent  measurements,  the 
older  values  being  data  compressed  and  stored  in  long-term 
memory.  The  TSMD  will  also  detect  sp)ecific  stress  profiles  that 
could  damage  equipment  and  note  their  occurrences  in  the  long 
term  memory.  When  stress  data  is  needed  by  either  Smart  BIT  or 
maintenance  equipment,  this  information  is  retrieved  from  the 
long  term  memory. 

When  a  failure  condition  is  detected  by  the  smart  BIT,  the  TSMD 
is  asked  to  return  relevant  stress  data.  Depending  on  the 
criticality  of  the  system  to  the  mission  and  flight  safety,  the 
Smart  BIT  will  continue  to  analyze  both  the  functional  test  data 
stream  and  the  TSMD  output.  If  necessary,  this  process  may  be 
performed  off-line  while  a  spare  unit  is  switched  in  place  of  the 
one  in  question.  If  a  decision  is  made  to  declare  a  unit  faulty, 
information  relevant  to  that  decision  process  will  be  stored  local 
to  that  unit's  nonvolatile  memory  for  access  later  by  other 
maintenance  processes.  (Ref  3) 

Fault  Logging  Using  Micro  TSMD 

The  next  step  in  the  field  validation  of  TSMD  for  enhancement  of 
onboard  BIT  performance  is  being  accomplished  under  the  Rome 
Laboratory  program  "Fault  Logging  Using  Micro  TSMD". 
Westinghouse  is  the  contractor  for  this  program  which  got 
underway  in  June  1991  and  is  scheduled  to  be  completed  in 
December  1993.  The  objective  of  this  program  is  to  adapt  a 
Micro  TSMD  type  design  to  log  environmental  stress  data  at  the 
time  of  occurrence  of  a  fault  in  a  selected  system  Line  Replaceable 
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Unit  (LRU)  and  demonstrate  these  capabilities  in  a  flight  data 
collection  program  using  a  fielded  Air  Force  System. 

The  LRU  selected  is  from  the  forward  radar  in  the  B-1B  bomber, 
this  LRU  has  a  past  history  of  a  significant  CND  rate  and  it  is 
believed  that  this  is  related  to  transient  environmental  conditions 
during  flight  Forty  B-1B  aircraft  will  be  equipped  with  Micro 
TSMD  devices  for  a  twelve  month  flight  data  collection  program. 
The  parameters  which  will  be  measured  by  the  Micro  TSMD  are 
temperature,  vibration  and  shock,  cooling  air  supply  pressure  for 
the  LRU  and  power  supply  conditions  for  the  LRU. 

The  environmental  inputs  are  placed  in  several  data  structures 
within  the  Micro  TSMD  which  include: 

1 .  Recent  Stress  Data 

2.  Historical  Stress  Data 

3 .  Peak  Stress  Data 

4 .  Fault  Window  Data 

The  largest  data  structure  is  the  Recent  Stress  Data  component. 
Two  (2)  hours  of  environmental  data  is  maintained  in  this  data 
structure.  The  environmental  data  includes:  temperature  data  from 
two  sensors  sampled  at  most  once  per  second,  voltage  data 
sampled  at  once  per  second,  FFT  information  calculated  from 
vibration  data,  shock  and  voltage  transient  information,  and 
power  on/off  events  all  with  the  necessary  information  to  relate 
this  data  to  real-time. 

Historical  Stress  Data  is  a  segment  of  the  nonvolatile  memory 
map  where  stress  information  from  the  Recent  Stress  Data 
component  is  compressed.  The  Micro  TSMD  was  designed  to 
manage  data  gathering  and  storage  in  excess  of  30  days; 
therefore,  the  Historical  Stress  Data  structure  was  developed  to 
maintain  information  with  less  resolution  until  the  Micro  TSMD 
could  be  debriefed. 

Peak  values  are  maintained  for  parameters  of  interest  such  as 
temperature,  voltage  level,  voltage  transient,  shock  magnitude, 
and  peak  power  measured  in  each  vibration  FFT  frequency  band. 
These  values  are  time  tagged  as  events  and  stored  in  nonvolatile 
memory. 

The  primary  processing  algorithm  of  the  Micro  TSMD  is  the 
Fault  Window  which  time  tags  environmental  stress  data  before, 
during,  and  after  a  fault  or  overstress  event  and  places  it  in 
nonvolatile  memory.  The  Fault  Window  consists  of  several  data 
structures  that  share  a  time  stamp  as  a  key  field.  If  a  system 
failure  is  detected  or  if  a  software  set  environmental  threshold  is 
exceeded,  the  data  resident  in  the  cyclical  buffer  is  time  tagged 
with  the  BIT  fault  code,  and  is  placed  in  nonvolatile  memory.  The 
Fault  Window  has  an  adjustable  interval.  Sampled  data  is  placed 
in  the  cyclical  buffer  at  frequencies  ranging  from  1  to  8,000  Hz. 
As  such,  when  a  fault  or  overstress  event  occurs,  the  cyclical 
buffer  is  halted.  Depending  on  the  width  of  the  Fault  Window 
which  is  software  programmable,  the  processor  counts  backward 
N  address  locations  and  places  N  words  of  cyclical  buffer  memory 
data  in  nonvolatile  memory.  The  same  process  occurs  for  data 
after  the  event.  Cyclical  buffer  memory  to  sustain  an  adjustable 
window  from  30  seconds  to  5  minutes  has  been  implemented  in 
the  Micro  TSMD.  (Ref  4) 

Each  time  that  the  host  LRU's  BIT  indicates  that  checkout  or 
removal  by  maintenance  personnel  is  required  after  a  mission,  the 
Micro  TSMD  will  be  debriefed.  Debrief  is  accomplished  through  a 
spare  test  connector  on  the  LRU  into  a  palm  held  computer.  The 
debriefed  Micro  TSMD  data  will  be  placed  in  a  data  base  with  all 
available  failure  and  maintenance  data  relative  to  each  indicated 
LRU  failure.  This  data  will  be  analyzed  in  an  attempt  to  correlate 
BIT  detected  failures,  the  type  of  failure  and  the  recorded 


environmental  data  surrounding  the  failure.  In  previous  Air  Force 
conducted  tests  there  were  indications  that  a  link  may  exist 
between  equipment  failures  and  environmental  stresses  for  this 
LRU.  The  twelve  month  data  collection  and  forty  aircraft  sample 
size  should  provide  a  statistically  stable  basis  for  correlation  of 
data. 

If  there  is  a  quantifiable  correlation  between  recorded  stress  data 
and  BIT  detected  failures  for  a  significant  number  of  occurrences, 
then  this  will  provide  a  sound  basis  for  further  development  and 
testing  of  onboard  TSMD/Smart  BIT  techniques.  Algorithms  can 
be  developed  for  onboard  testing  which  could  significantly  reduce 
the  number  of  indicated  failures  during  operation.  Currently  a 
program  that  would  conduct  a  requirements  analysis  phase  for 
integration  and  field  testing  of  intelligent  test/Smart  BIT 
techniques  with  TSMD  technology  is  being  initiated. 

CONCLUSIONS 

TSMD  technology's  feasibility  has  been  demonstrated.  The 
TSMD  module  showed  that  a  digital  environmental  measurement 
and  recording  device  could  be  developed  which  provided  very 
useful  information  in  a  field  environment  within  the  "lemory, 
processing  and  data  compression  constraints  of  current 
technology.  The  Micro  TSMD  is  a  significant  leap  forward  in 
physical  size,  which  permits  its  use  on  circuit  cards  in  LRUs  or 
Line  Replaceable  Modules  (LRMs),  and  enhanced  operational 
performance.  This  enhanced  performance  permits  greater 
precision  in  data  measurement  and  recording  and  the  potential  for 
its  integration  ir.to  the  BIT  methodology  for  the  host  LRU. 

"Fault  Logging  Using  Micro  TSMD"  is  a  critical  step  in  the 
development  of  future  onboard  BIT  performance  enhancements.  It 
should  provide  the  evidence  that  many  BIT  detected  failures  are 
not  hard  failures,  but  soft  failures.  TTiis  means  system 
performance  was  temporarily  out  of  tolerance  due  to  transient 
anomalies  in  some  aspect  of  the  operational  environment.  Also, 
it  should  indicate  those  intermittent  hard  failures  which  may  be 
activated  only  by  a  specific  environmental  stimulus. 

Concurrently  with  the  development  of  the  TSMD  technology. 
Smart  BIT  techniques  have  been  investigated  and  developed.  All 
the  technology  is  in  place  and  the  plans  made  for  a  major  jump  in 
the  performance  of  an  onboard  diagnostic  capabilities.  The 
benefits  are  fewer  pilot  distractions  from  false  BIT  indications, 
better  aircraft  availability  and  reduced  maintenance  costs. 
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Discussion 

QUESTION  R. LITTLE 

Could  you  give  details  on  the  number  and  location  of  TSMD  sensors 
required  to  cover  a  complete  avionics  system? 

REPLY 

Several  alternatives  have  been  considered.  The  current  micro  TSMD  (2.5 
*  5  cm)  is  too  large  to  fit  onto  every  card.  Currently  one  device  per  box 
or  LRU  is  being  utilised  with  up  to  4  temperature  sensors  and  1 
accelerometer  which  can  be  either  internal  or  external  to  the  TSMD. 
Suggested  locations  for  the  sensors  include  input  and  output  cooling  air 
temperature,  any  thermal  problem  areas  and  LRU  chassis  acceleration  normal 
to  the  circuit  cards. 

There  are  current  plans  to  make  the  micro  TSMD  even  smaller 
(approximately  2.5  *  2.5  cm)  so  that  it  becomes  more  feasible  to  put  one 
on  every  circuit  card  or  LRU.  These  TSMD's  will  have  the  capability  for 
additional  external  sensors  and  more  flexibility  for  communicating  with 
system  diagnostics. 
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SUMMARY 

In  view  of  ever  decreasing  budgets,  the  Aspect  Life  Cycle 
Cost  (LCC)  of  a  product  is  gaining  in  importance. 

Telefunken  Systemtechnik  has  been  working  for  more  than  10 
years  in  the  area  of  integrated  Logistic  Support  with  the  aim  of 
minimizing  the  LCC  for  new  products.  TICAMS  is  one  result 
of  this  work. 

TICAMS  is  a  general  test  system  for  development,  production 
and  utilization  phases.  This  universal  concept  automatically 
provides  the  practical  and  theoretical  knowledge  of  each  phase 
whilst  minimizing  costs.  In  doing  so,  the  overall  system 
requirements  with  regard  to  the  user  are  supported  by 
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TICAMS  thus  leading  to  cost  minimization  in  the  utilization 
phase. 

TICAMS  supports  the  user  in  conducting  maintenance  and 
repair  work,  optimizes  essential  tests  and  diagnosis  procedures 
during  the  test  procedure,  and  supports  the  user  during  each  of 
the  work  stages  by  means  of  a  comprehensive  user  interface 
and  information  system. 

TICAMS  uses  test  strategies  specified  to  the  product  which  is 
to  be  tested  and  maintained,  and  works  with  an  expert  system 
for  optimizing  the  diagnosis  and  test. 
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TICAMS  works  with  an  integrated  instruction  and  user- 
guidance  concept  which  facilitates  a  comprehensive  training 
program  -  from  industrial  training  or  on-site  user  training  to 
regular  refresher  courses  for  the  user. 

Therefore  TICAMS  uses  electronic  documentation.  This 
electronic  documentation  is  partly  conveyed  in  the  form  of 
video  sequences,  thus  reducing  the  costs  required  to  compile 
the  documentation. 

TICAMS  was  developed  according  to  the  principle  that  only  a 
single  test  system,  which  covers  all  of  the  user's  requirements 
and  is  able  to  be  networked  to  other  systems,  can  reduce  the 
LCC. 


CONTENTS 

1.  INTRODUCTION 

2.  OBJECTIVE  OF  TICAMS 

2.1  Repair  Support 

2.2  Optimization  of  Test  Procedures 

2.3  Support  by  Means  of  Technical 

Publications 

2.4  Comprehensive  Training  Program 

2.5  Phase  Overlapping 

3.  FEATURES  OF  TICAMS 

3.1  Graphics  User  Interface 

3.2  User  Guidance 

3.3  Strategies 

3.3. 1  Test  Strategies 

3.3.2  Diagnosis  Strategies 

3.4  Expert  Systems 

3.5  Electronic  Documentation 

3.5. 1  Document  Types 

3.5.2  Interfaces  for  Documentation  Tools 

3.5.3  Video 

3.6  Evaluating  Results 

3.6.1  Reports 

3-6.2  Statistics 

3  6. 3  Quality-Assurance  Systems 

3.7  Networking 

4.  STRUCTURE  OF  TICAMS 

5.  TICAMS  IMPLEMENTATIONS 

5.1  Tornado  Self-Protection  Jammer  Tester 

5.2  Eurofighter  90  Radar- Production  Tester 

5.3  Growth  Potential 

6.  CONCLUSION 


LIST  OF  ABBREVIATIONS 

AITEST  Artificial  Intelligence  Test,  a  product 

of  Intelligent  Electronics  Ltd. 


CIROS 

Computer-Aided  Integrated 
Documentation  RetrievalSystem  on 
Optical  Storage  Media,  a  Product  of 
Messerschmitt-Bolkow-Blohm 

GUI 

Graphical  User  Interface 

Hyperdoc 

Hyperdocumentation,  a  Product  of 
GECI  International,  S.A. 

ILS 

Integrated  Logistic  Support 

LCC 

Life  Cycle  Cost 

MBB 

Messerschmidt  Bolkow  Blom 

TICAMS 

Telefunken  Integrated  Computer- 
Aided  Maintenance  System 

UUT 

Unit  Under  Test 

vss 

Video  Support  System,  a  Product  of 
Telefunken  Systemtechnik 

1.  Introduction 

Maintenance  is  a  term  which  is  of  particular  significance 
today.  The  meaning  of  this  concept,  however,  largely  depends 
on  personal  interpretation  and  understanding.  This  is  due  to 
the  variety  of  individual  activities  and  responsibilities  and 
their  specific  motives  and  objectives.  Interest  or  disinterest  in 
this  area  varies  considerably,  from  the  purely  operational 
motive  and  the  financial  aspect  to  personal  motives  on  the 
customer  side  as  well  as  on  the  manufacturer  side.  The  main 
concern  of  all  parties  involved  is  to  protect  their  own  interests. 

There  is  only  one  means  of  balancing  these  different 
considerations  and  this  is  in  the  form  of  dialogue.  A 
cooperative  dialogue  between  all  partners  involved  is 
necessary  to  pursue  the  objective  of  minimizing  the  cost  factor 
for  the  system/equipment  utilization  phase. 

This  dialogue  must  be  started  in  the  initial  system-definition 
phase  since  the  maintenance  strategy,  with  all  of  its  positive 
and  negative  aspects,  is  already  being  devised  in  this  early 
cycle  phase. 

The  growth  in  technology  and  complexity  of  equipment  has 
increased  the  need  to  use  excellent  teams  consisting  of  a  large 
number  of  specialists  to  plan  how  the  equipment  is  to  be  put 
into  operation.  The  ability  to  provide  self-help  with  regard  to 
repair  and  re-commissioning  work  has  had  to  be  reduced. 

The  technical  developments  have  only  gradually  allowed 
technicians  to  exchange  complete  units  instead  of  repairing 
them.  This  has  led  to  an  extreme  need  for  capital  investment 
due  to  higher  numbers  of  turn  around  units. 

This  awareness  of  higher  capital  requirements  initially  had  no 
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direct  consequences.  There  was  sufficient  money  in  the 
budgets.  Today  monetary  conditions  have  deteriorated  in  a 
dramatic  manner.  In  order  to  be  able  to  introduce  and  to  utilize 
the  necessary  equipment,  not  only  the  procurement  costs  are 
carefully  considered  today  but  the  "follow  on  cost  of  service 
life"  also  plays  an  important  part. 

The  use  of  "Integrated  Logistic  Support"  resolves  the 
problems  listed  above  in  that  the  entire  effectiveness  of  the 
product  must  not  only  be  defined  according  to  its  operational 
requirements  but  also  to  the  degree  of 

-  reliability 

-  test  characteristics 

-  maintenance  characteristics 

involved.  These  logistic  design  elements  largely  determine 
the  entire  life  cycle  costs  of  the  equipment. 

Requirements  regarding 

•  maintenance  personnel  and  skill  level 

-  test  equipment 

-  spare  parts 

-  repair  and  overhaul  work 

directly  depend  on  the  logistic  engineering  process  and  its 
results. 


Our  Telefunken  Systemtechnik  Integrated  Logistic  Support 
department  guarantees  the  integration  of  logistic  engineering 
and  support  elements  into  the  system  design.  They  are  already 
equal  partners  to  system  development  within  the  concept, 
definition  and  early  design  phase. 

With  more  than  10  years  experience  within  the  field  of  logistic 
engineering  our  products  are  designed  under  the  scope  of 
performance  and  logistic  supportability  with  equal  priority  to 
reducing  life  cycle  cost. 

Figure  1  - 1  shows  the  development  cycle  of  an  1LS  product. 
The  result  is  a  logic  support  system  which  is  designed 
according  to  logistic -engineering  requirements. 

Telefunken  Integrated  TICAMS  is  a  logistic  support 
system 


FIGURE  1-1  TICAMS  IS  AN  ILS  PRODUCT 
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2,  Objective  of  TICAMS 

Figure  2-1  shows  indeed  that  the  major  part  of  the  total  LCC  is 
concentrated  in  the  production  and  series  phases,  the  type  and 
occurence  of  these  costs  are  however  established  in  the  initial 
development  phases  of  a  pnxh’ct. 

The  primary  objective  of  TiCAMS  is  to  minimize  the  costs  in 
the  utilization  phase. 

Figure  2-2  shows  in  an  example  of  Test  and  Repair  Activities 
the  variety  of  activities  which  must  be  performed  by  the  user 
during  this  task.  These  activities  are  extremely  personnel¬ 
intensive  and  therefore  cost-intensive. 

TICAMS  is  implemented  at  this  point  and  supports  the  entire 
process  during  the  utilization  phase.  Furthermore,  TICAMS 
provides  feedback  to  the  previous  development  phases. 

2.1  Repair  Support 

Repair  work  usually  requires  access  to  different  software 
repair  systems,  e.g.: 

configuration  control  for  the  equipment 

access  to  warehouses  for  spare  parts  and  to  check  their 
availability 

ordering  of  spare  parts  for  conducting  repair  work 

checking  of  the  service  life  of  the  system 

entering  the  actual  data  for  the  system  in  quality- 
assurance  systems 

The  objective  of  TICAMS  is  to  provide  all  of  the  essential 
repair  work  by  means  of  the  test  system. 


%Oagrwal 

Dtttorminalion  %  Degree  of 

of  Tola*  LCC  Total  LCC 


Figure  2- 1  Life  cycle  costs  2050s  aw 


2.2  Optimizing  the  Test  Procedures 

Tests  are  usually  performed  on  the  equipment  while  the  repair 
work  is  being  carried  out.  These  tests  can  also  be  extremely 
costly  and  time-consuming  and  may  end  in  diagnostic 
routines. 

TICAMS  aims  to  optimize  the  complete  test  procedure, 
(including  any  necessary  diagnostic  routines)  with  regard  to 
the  time/cost  aspect,  therewith  taking  into  consideration  the 
specific  features  of  the  equipment. 
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FIGURE  2-2  TESTER-relevant  activities  during  the  utilization  phase 


2.3  Support  by  Means  of  Technical  Publications 

Technical  publications  (i.e.  documentation)  arc  required  for  all 
activities  with  the  equipment.  These  include  the  following: 

functional  description  of  the  equipment 
instruction  manuals 
maintenance  manuals 
repair  manuals 

spare  parts  catalogues  (illustrated  with 
text) 

This  documentation  is  required  for  both  the  equipment  and  the 
test  and  support  systems.' 

The  problem  for  the  user  is  that  he  must  always  have  this 
documentation  at  hand  if,  due  to  overall  complexities,  he 
requires  certain  information  and  secondly,  he  must  know 
where  to  find  the  missing  information. 


TICAMS  aims  firstly  to  provide  the  user  with  all  of  the 
required  information  in  the  form  of  electronic  documentation. 
Secondly,  the  user  should  be  presented  with  the  well-ordered 
relevant  information  for  the  respective  process.  Thirdly, 
TICAMS  aims  to  present  this  comprehensive  information  to 
the  user  not  in  a  I  :  1  ratio  but  with  a  higher  information 
density  (e.g.  display  of  logically  annotated  photographs  or 
videos).  This  represents  the  sole  means  of  reducing  the  total 
documentation  costs. 

2.4  Comprehensive  Training  Program 

Every  user  of  equipment  or  test  system  must  receive 
appropriate  training  and  instruction. 

This  instruction  or  training  must  take  into  consideration  the 
skill  level  of  the  trainee  and  the  activities  which  he  will 
subsequently  perform.  Depending  on  the  type  of  training 
required,  this  will  either  take  place  in  an  industrial  centre  or 
directly  with  the  user. 


The  high  cost  of  compiling  this  documentation  poses  a  further 
problem. 


Additional  specific  training  material  is  often  compiled  to 
conduct  these  instruction  and  training  courses. 
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TICAMS  aims  to  provide  the  user  with  an  optimum  us>  r 
guidance  system  for  the  test  system  and  to  supply  all  of  the 
essential  information  for  the  respective  process  directly  to  the 
concerned  position.  T1CAMS  will  provide  this  information 
independent  of  the  skill  level  of  the  user.  An  additional 
training  mode  is  intended  to  provide  the  user  with  any 
information  beyond  the  scope  of  the  respective  process  and 
enable  him  to  access  the  required  instructions  for  the  test 
system  and  equipment. 

These  facilities  can  be  used  for  initial  training  purposes  and 
later,  as  a  refresher  course. 

Training  documentation,  therefore,  can  be  kept  to  a  minimum 
or.  in  some  cases,  may  not  be  required  at  all 

2.5  Phase  Overlapping 

As  described  in  sections  1  and  2.  a  product  must  be  regarded 
in  terms  of  phases. 


The  testing  and  maintenance  aspects  cannot  be  taken  into 
account  during  the  production  phase  or  later  with  the  user. 

The  aim  of  T1CAMS  is  to  examine  in  an  initial  development 
phase  the  testability  of  the  machines  by  using  rapid  prototypes 
and  knowledge  from  expert  systems.  Weak  points  can  thus  be 
easily  eliminated  when  the  machine  is  being  designed. 

Furthermore,  by  implementing  TICAMS  in  the  production 
process,  important  characteristics  of  the  equipment  can  be 
collected  and  incorporated  into  the  test  system.  This 
information  can  then  be  supplied  on-site  to  the  TICAMS  user. 

On  the  other  hand,  by  incorporating  report  and  analysis 
systems,  any  weak  points  in  the  machine  can  be  detected  and, 
by  networking  the  systems,  can  lead  to  design  changes  thus 
constituting  a  full  cycle  (see  figure  2-3). 


utilisation  phase 


function 

and 

diagnose  tests 


n> 


production  phase 


Figure  2-3  TICAMS  Phase  Overlapping  between  Production  and  Utilization 
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3.  Features  of  TICAMS 

3. 1  Graphics  Iser  Interface 

TICAMS  features  a  state-of-the-art  Graphics  User  Interface 
The  system  can  currently  be  run  under  UNIX  (X- 1 1  window) 
All  of  the  Graphics  User  Interfaces  have  been  created  usinc 
the  OSF  Motif  and  can  be  operated  with  a  mouse.  The  me:  u 
system  features  both  pop-up  and  pull-down  menus  The  er  .ire 
Graphics  User  Interface  (GUI)  is  of  modular  design  and  thus 
can  be  easily  modified  to  meet  particular  customer 
requirements. 

When  the  system  is  initialized,  the  Graphics  User  Interfaces 
are  configured  automatically,  according  to  the  user's 
requirements.  The  selected  configurations  are  taken  from 
resource  files.  This  also  facilitates  subsequent  adaptation  for 
new  software  tools,  such  as  electronic  documentation  tools  for 
example  (see  section  3.5).  In  this  way,  the  Video  Support 
System  (VSS),  for  example,  can  be  integrated  subsequently 
into  the  system  by  simply  modifying  the  resource  file.  The 
existing  system  therefore  does  not  have  to  be  modified. 

3.2  User  Guidance 

One  of  the  main  problems  with  complex  tesi  systems  is  that 
they  are  difficult  to  operate. 

TICAMS  provides  the  user  with  a  comprehensive  user 
guidance  system  using  Graphics  User  Interfaces  (GUI). 

After  the  test  system  has  been  started  automatically,  the  user  is 
presented  immediately  with  a  log-in  GUI.  When  logging  in, 
the  user  is  assigned  a  skill  level;  this  enables  the  degree  of 
support  the  user  will  receive. 

A  total  of  3  different  levels  is  recommended  in  practice. 

master  level:  for  the  experienced  user  who 
requires  minimum  support 

assistant  level:  for  the  average  user  who  has 
received  the  standard  training 

beginner  level:  for  the  beginner  who  has  yet 
to  receive  training  for  the 
test  system. 

Each  user  can  select  a  level  lower  than  his  assigned  level  when 
logging  in,  so  that  users  who  have  not  worked  with  the  test 
system  fora  considerable  length  of  time  can  reacquaint 
themselves  quickly. 

TICAMS  can  be  supplied  in  two  versions 
Development  Version 


Unlike  the  user  in  the  Development  Version,  the  user  in  the 
Runtime  Version  does  not  have  access  to  the  operating 
system.  All  of  the  functions  within  the  test  system  can  only  be 
operated  sia  GUI  Similarly,  the  GUI  cannot  be  manipulated. 
Esen  the  classification  and  administration  of  the  users  who 
have  he  authorized  to  log  in  are  performed  via  GUI. 

In  the  Run  me  Version  this  prevents 

manipulations  of  the  system 
incorrect  operation  of  the  test  system. 

After  the  user  has  logged  in.  he  can  access  all  of  the  functions 
of  the  test  system,  such  as 

user  administration  (provided  that  the  user 
is  authorized  to  do  so) 

system  configuration  (provided  that  the 
user  is  authorized  to  do  so) 

data  bank  for  previous  test  (test  history) 

reporting  system 

self-test  and  automatic  calibration  of  the 
test  system 

specific  tests 

electronic  documentation 

The  user  is  guided  within  each  function  by  means  of  the  GUI 
from  which  he  can  select  subfunctions.  For  example,  he  can 
view,  edit,  print  or  create  backups  of  the  information  in  the 
test  history  data  bank  for  all  or  just  certain  test  subjects.  The 
same  facilities  are  available  in  the  reporting  system. 

Naturally,  backup  copies  can  be  reloaded  into  the  system  so 
that  older  information  can  be  reviewed  on  all  of  the  test 
systems  at  any  time. 

The  user  receives  continuous  guidance  during  the  test  in  order 
to  perform  the  test  and  execute  the  corresponding  commands. 
An  appropriate  set  of  help  texts  can  be  selected  for  each 
process  from  the  electronic  documentation. 

Each  user  can  call  the  electronic  documentation  immediately 
after  he  has  logged  in.  He  can  thus  access  all  of  the 
configured  documentation  tools  and  the  corresponding 
documentation.  In  this  training  mode,  the  user  can  view  the 
documents  freely  on  the  screen. 

3.3  Strategies 

The  term  strategies  is  used  here  to  describe  the  various  ways 
in  which  a  function  test  or  diagnostic  procedure  can  be 
performed  if  a  fault  occurs.  A  distinction  is  made,  therefore 
between  a  test  strategy  and  a  diagnostic  strategy. 


Runtime  Version 
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3.3.1  Test  Strategy 

All  of  the  test  programs  in  TICAMS  are  of  modular  design.  A 
test  program  consists  of  test  modules  which  in  turn  comprise 
individual  test  sequences.  Each  test  sequence  is  a  complete 
measurement  conducted  on  a  test  subject  and  thus  represents 
the  smallest  test  which  can  be  performed. 

This  structure  enables  the  user  to  choose: 

to  perform  a  complete  test  program  (using 
all  of  the  test  modules) 

to  perform  an  individual  test  module. 

The  user  can  then  select  various  evaluation  modes  within  these 
different  test  modes 

display  all  test  results  on  the  screen 

only  display  test  results  on  the  screen  if  a 
fault  occurs 

display  none  of  the  test  results  on  the 
screen 

In  addition,  the  user  can  define  a  break  point  to  change  the 
evaluation  mode.  He  can  also  form  group  tests  for  individual 
test  modules  in  the  test  mode. 

All  of  the  test  modules  in  each  of  these  groups  are  performed 
in  sequence. 

If  no  faults  are  found  with  the  specimen,  further  strategies  are 
not  required.  The  user  can  call  individual  test  modules  as 
required  in  order  to  perform  individual  tests.  He  must  only 
ensure  that  a  complete  test  program  is  performed  once  so  that 
the  specimen  can  be  marked  as  "GO",  i.e.  fault  free. 


Since  the  specimen  must  be  released  as  "fault  free”  as  quickly 
as  possible,  for  cost  and  time  reasons,  the  "complete  test 
program"  is  used  as  the  default  test  mode  in  the  test  system. 

If  a  fault  is  found  on  the  test  subject,  however,  the  time 
required  until  the  subject  can  be  released  after  it  has  been 
repaired  must  be  taken  into  account.  In  doing  so,  it  is 
necessary  to  take  into  consideration  that  a  diagnostic  result  for 
complex  modules  is  not  always  100  %  accurate  (for  example  a 
diagnosis  result  could  show  that  one  specimen  module  is 
responsible  for  60  %  of  the  fault  and  another  for  40  %). 

If  the  complete  test  program  is  then  performed  after  a  module 
has  been  replaced  in  the  specimen,  the  fault  may  still  be 
present.  In  this  case,  the  "wrong"  module  was  replaced  and  a 
complete  test  program  was  performed  in  vain.  In  order  to 
overcome  this  problem,  TICAMS  provides  the 

Verification  Test 

TICAMS  starts  automatically  at  a  test  sequence  which  should 
have  changed  from  "Fail"  to  "Pass"  after  the  module  has 
successfully  been  replaced.  If  this  is  the  case,  TICAMS  starts 
the  complete  test  program  automatically.  If  the  fault  is  still 
present,  TICAMS  starts  the  next  diagnostic  routine 
automatically.  TICAMS  generates  the  appropriate 
information  for  the  verification  test  from  the  data  bank  of 
previous  tests. 

3.3.2  Diagnostic  Strategies 

There  are  several  ways  of  locating  a  fault  in  the  test  subject 
once  it  has  been  detected  (see  Fig.  3-1). 


Figure  3-1  Diagnose  structures 
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The  conventional  method  used  to  perform  a  diagnostic  routine 
is  to  construct  a  diagnosis  tree  on  which  each  test  result  is 
represented  by  a  node.  There  are  only  discrete  decision 
branches,  e.g.  yes/no,  pass/fail,  true/false,  etc.  According  to 
the  probability  theory,  these  decisions  are  100  %  accurate. 
Consequently,  all  of  the  possibilities  must  be  taken  into 
account  when  constructing  the  diagnosis  tree.  Only  these 
predefined  decisions  produce  a  diagnostic  result. 

This  diagnostic  procedure  generally  does  produce  an 
extremely  satisfactory  result  for  non-complex  modules  with  a 
small  number  of  test  points  which  can  be  accessed  easily.  The 
disadvantage  with  this  method,  however,  is  that  subsequent 
modifications  can  require  considerable  time  and  expense 

Another  method  of  performing  a  diagnostic  routine  is  to  ask 
after  every  intermediate  test  result  which  test  will  increase  the 
probability  that  a  hypothetical  fault  location  will  be  confirmed. 
This  method  is  based  on  probability  analysis.  The  advantage 
thereby  is  that  the  rigid  test  sequences  are  replaced  by  a  free 
test  pool.  The  procedure  is  only  generated  by  a  concrete  fault. 
A  further  advantage  of  this  method  is  that  it  can  be  expanded 
as  required  and  also  takes  into  consideration  any  improbable 
fault  types  in  the  test  subject.  The  required  programming  is 
performed  using  an  expert  system. 

As  a  further  strategy,  it  is  necessary  to  define  the  size  of  the 
individual  elements  in  the  test  pool.  The  module  structure  of 
the  TICAMS  test  program  offers  two  ways  of  doing  just  this. 
Firstly,  a  test-pool  element  can  be  defined  as  a  test  module. 
Secondly,  a  test-pool  element  can  be  defined  as  a  test 
sequence.  It  is  not  possible  to  say  at  the  outset  which  of  these 
methods  will  be  less  time  consuming.  A  test  sequence  is 
indeed  specific  but  the  measurement  must  be  initialized  for 
each  sequence.  If  several  test  sequences  are  performed 
consecutively,  the  required  time  may  be  considerably  greater 
than  with  one  test  module,  in  which  a  rigid  set  of  test 
sequences  is  defined  and  which  consequently  can  be  included 
in  the  previous  initialization. 

A  further  criterion  which  must  be  considered  is  the  point  at 
which  the  system  switches  to  the  diagnosis  branch  after  a  fault 
has  been  detected.  In  a  diagnosis  tree,  the  system 
automatically  continues  in  the  tree  structure  after  the  first  fault 
has  been  detected.  This  requires  first  that  all  of  the  previous 
measurements  have  also  been  performed,  i.e.  the  diagnosis 
procedure  can  only  be  conducted  from  a  "complete  test 
program”  mode.  Secondly,  test  modules  or  sequences  which 
have  not  yet  been  performed  but  whose  results  are  required 
first  in  the  diagnostic  tree  must  be  incorporated  into  the  test 
program  several  times. 

These  factors  are  not  required  with  a  free  test  pool.  A 
diagnosis  procedure  can  be  initiated  before  any  test  module. 
Furthermore,  each  test  module/sequence  is  only  required  once 
by  the  system. 

TICAMS  transmits  each  test  result  to  the  expert  system 
continuously  so  that  the  latter  is  always  informed  of  the 
current  test  status.  If  a  fault  occurs,  the  expert  system  assumes 
control  over  the  remaining  test  procedure. 


In  order  to  perform  a  diagnosis  procedure  with  optimum  time 
requirements  for  each  specimen,  the  following  can  be  selected 
via  resource  files  and  modified  if  necessary: 

diagnostic-tree  structure 
diagnosis  expert  system 
test  selection  via  test  module 
test  selection  via  test  sequence 

3.4  Expert  Systems 

3.4.1  General 

Artificial  intelligence  is  an  area  given  to  considerable  research 
and  development  throughout  the  world.  The  sub-field  of 
expert  systems  is  thus  extremely  advanced  and  has  stabilized 
to  such  an  extent  that  good  software  tools  are  now  available 
for  the  special  applications  which  are  common  to  electronic 
diagnostic  procedures. 

These  diagnostic  tools,  however,  have  been  largely  designed 
to  meet  only  the  practical  requirements  of  the  user.  In  general, 
these  tools  are  simply  shells  for  knowledge-based  systems 
(KBS).  The  test  engineer  must  first  familiarize  himself  with 
these  shells  before  they  can  be  applied.  Furthermore,  in  order 
to  create  applications  in  these  shells,  specific  artificial 
intelligence  (AJ)  languages  such  as  PROLOG,  LISP,  or 
SMALLTALK  are  generally  required.  These  rule-based 
systems  therefore  require  that  each  specimen  problem  be 
translated  into  one  of  these  special  AI  languages.  This  is 
generally  performed  by  so-called  application  engineers  who 
are  specialists  in  the  relevant  KBS  and  its  AI  language. 
Incorporating  specimen  information  into  the  KBS,  therefore, 
requires  at  least  one  intermediate  stage. 

In  addition,  the  interfaces  of  these  KBS  tools  are  not  designed 
to  meet  field  or  shop  requirements  and,  therefore,  must  be 
expanded  accordingly  -  a  process  which  requires  considerable 
time  and  effort.  Furthermore,  rule-based  systems  with  an 
automatic  learning  mode  have  not  yet  bee.)  brought  on  to  the 
market. 

3.4.2  Requirements 

In  view  of  the  reasons  listed  above,  Telefunken  Systemtechnik 
has  compiled  a  list  of  basic  requirements  which  must  be 
satisfied  by  an  expert  system: 

model-based  system 

easy  to  handle  by  test  engineers 
(without  specific  knowledge  of  the 
internal  software  structure) 

no  need  to  learn  specific  AI  language 

assistance  to  implement  testability  in  the 
UUT 

on-line  verification  of  diagnosis  results 
automatic  learning 
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convenient  man/machine  interface  for 
operator  and  developer 

configurable  for  different  diagnostic 
strategies 

fully-automatic  mode 

TICAMS  provides  a  standard  interface  for  connecting  an 
expert  system  so  that  the  latter  is  freely  interchangeable. 

3.4.3  A1TEST 

Taking  the  basic  requirements  listed  above  into  consideration, 
Telefunken  Systemtechnik  has  selected  the  expert  system 
ATTEST  and  implemented  it  in  TICAMS. 

ATTEST  has  been  modified  in  accordance  with  the  basic 
requirements  of  Telefunken  Systemtechnik  and  is  now 
available  in  the  revised  version. 

ATTEST  functions  as  a  module-based  system.  The  UUT 
module  is  entered  in  the  system  in  the  form  of  an  extended 
block  diagram.  The  system  features  an  internal,  universal 
knowledge  base  in  which  the  basic  principles  of  electronic 
diagnosis  are  stored,  i.e.: 

the  structure  of  a  block  diagram 

the  function  of  electrical  switching 
elements 

basic  understanding  of  electrical 
measurements. 

Figure  3-3  shows  the  three  phases  in  which  a  UUT-specific 
knowledge  base  can  be  created,  processed  and  modified. 


In  this  respect,  AITEST  is  equipped  with  the  following 
features: 

During  the  test-design  phase 

test-point  optimization  from  the  test  point 
of  view 

test  documentation  based  on  the 
Acceptance  Test-Procedure  level 

test  design  tools  (simulation  of  test 
scenarios)  to  support  table-top  test 
program  (TP)  development 

debugging  aids  for  the  Test  Programs 

automatic  generation  of  Test  Program- 
control  flow 

During  the  test-execution  phase  (see  also  Figure  3-2) 

comprehensive  hierarchical  diagnostic 
assessment 

cost-effective  dynamic  test  sequencing 
communication  with  the  test  system 
test  data  logging  into  archive  files 
During  the  post-test  phase 

data  collection  from  different  sites 

automatic  learning  of  diagnostic 
knowledge 

statistical  and  data  analysis. 


Figure  3-2  Effect  of  the  expert  system  on  the  test  sequence 
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AITEST  offers  tools  across  all  Test  Phases: 


1.  PHASE 
TEST  DESIGN 


2.  PHASE 
TEST  EXECUTION 


3.  PHASE 
POST-TEST 
ANALYSIS 


KB  -  Knowledge  Base 
KC  ^  Knowledge  Compilation 


Figure  3-3  AITEST  architecture 
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AITEST  was  implemented  in  TTCAMS  based  on  the  afore 
mentioned  features  and  is  available  as  a  diagnostic  tool  for  all 
test  specimens.  The  post-test  features  of  AITEST  are  only 
used  at  one  central  location  to  facilitate  central  software 
configuration  control  over  the  UUT  knowledge  bases. 

The  automatic  learning  mode  is  implemented  specifically. 
Each  learning  procedure  is  based  on  statistical  calculations 
which  can  be  used  to  modify  the  probability  values  in  the 
knowledge  bases  so  that  more  accurate  diagnostic  results  can 
be  obtained.  New  specimen  information  or  "real-life 
experience"  from  the  production  or  application  processes  can 
be  easily  implemented  in  the  system. 


Each  learning  procedure  is  thus  fully  reversible.  The  internal, 
statistical  calculations  guarantee  a  convergence  versus 
stability,  even  with  different  learning  curves. 

3.5  Electronic  Documentation 

Normally  a  whole  range  of  technical  publications  in  the  form 
of  handbooks  or  microfiches  is  available  for  operating, 
maintaining,  repairing,  checking  and  calibrating  the  test 
system  as  well  as  for  the  test  specimen  itself.  Each  of  these 
technical  publications  represents  a  source  of  information  for 
the  user.  Since  data  processing  is  nothing  but  a  means  of 
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processing  information,  it  seems  logical  to  store  these 
publications  on  electronic  media,  i.e.  as  electronic 
documentation.  The  problem,  however,  is  a  matter  of  user 
acceptance,  as  no-one  likes  to  read  a  text,  page  for  page,  from 
a  screen.  Diagrams,  graphs  etc.,  however,  are  more 
acceptable.  This  is  undoubtedly  due  to  the  fact  that  they 
contain  a  higher  information  content  on  a  smaller  area.  Hence 
the  saying:  "a  picture  paints  a  thousand  words!". 

The  solution  to  this  problem,  therefore,  is  to  convert  as  much 
of  the  text  as  possible  to  diagrams,  graphs  etc. 

When  implementing  the  documentation  in  TICAMS, 
considerable  importance  was  attached  to  the  fact  that  specific, 
electronic  documentation  for  each  individual  operation  should 
be  available  to  the  user,  both  in  its  general  form  (training 
mode)  and  as  part  documentation,  as  relevant  for  the  various 
work  stages  (on-line  mode). 

3.5.1  Document  Types 

Depending  on  the  type  of  technical  publication,  it  can  either  be 
displayed  directly  (1 : 1)  on  the  screen  or  must  first  be 
converted  accordingly. 

Conversion,  in  this  sense,  means  that  the  publication  is  not 
compiled  in  the  conventional  manner  but  is  first  adapted 
accordingly. 

The  following  is  a  list  of  the  different  types  of  technical 
publication  with  the  corresponding  type  of  display  on  the 
screen. 

Configuration  lists  are  generally  present  in  data¬ 
bank  structure  and  can  thus  be  easily  transferred 

Operating  manuals  are  usually  in  text  form  and 
therefore  must  be  converted 

Maintenance  manuals  are  usually  in  text  and 
illustrated  form;  conversion  is  recommended 

Repair  manuals  are  generally  in  illustrated  form 
and  are  therefore  ideal  for  conversion  to  videos 

Functional 

descriptions  are  mostly  in  text  form.  Except 

in  an  abbreviated  form,  it  is  not 
recommended  to  supply  this 
information  on  the  screen 


Spare  parts  catalogues, 
illustrated  parts 

catalogues  are  present  in  text  and 

illustrated  form.  The  text  is  in 
data-bank  structure.  The  entire 
representation  can  be  displayed 
on  the  screen. 

3.5.2  Interface  for  Documentation  Tools 

TICAMS  provides  standard  interfaces  for  the  documentation 
tools.  The  modular  and  freely  configurable  Graphics  User 
Interfaces  (see  section  3.1)  enable  new  documentation  tools  to 
be  entered  at  any  time. 

At  present,  the  existing  TICAMS  implementations  have 
interfaces  to  the  following: 

CIROS 

Hyperdoc 

Video  Support  System 

The  interfaces  are  designed  in  such  a  way  that  the 
documentation  can  be  called  via  the  Graphics  User  Interface 
both  as  general  documentation  (training  mode)  as  well  as 
specific  part  documentation  for  individual  procedures. 

3.5.2. 1  CIROS 

The  CIROS  system  facilitates  the  use  of  an  illustrated  parts 
catalogues,  spare  parts  catalogues  as  well  as  a  configuration 
control.  Furthermore,  access  to  the  warehouse-management 
and  ordering  systems  can  also  be  implemented  via  CIROS. 

3. 5.2.2  Hyperdoc 

Hyperdoc  facilitates  the  use  of  brief  descriptions  and  block 
diagrams. 

3. 5.2. 3  Video  Support  System 

A  whole  range  of  technical  publications  can  be  displayed 
effectively  on  the  screen  by  means  of  the  video  support 
system  Typical  application  areas  include: 

disassembling/reassembling  the  UUTs 

connecting/disconnecting  the  UUT  to/ 
from  the  test  system 

manual  intervention  during  the  test  (e.g. 
manual  measurements,  adjustments  etc.) 


Block  diagrams  are  in  illustrated  form  and 
therefore  can  be  easily  recreated  on  the  screen  as  brief 
information 

Schematic  diagrams  are  generally  in  large  format.  It 
is  not  recommended  to  recreate 
these  on  the  relatively  small 
screens. 


module-change  activities 
calibration  activities 
self-test  activities 

training  activities  for  handling  the  test 
system 

UUT  training  (function,  mechanical 
layout,  maintenance  activities  etc  ) 
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All  of  the  videos  in  the  video  support  system  are  of  modular 
structure.  A  variety  of  video  sequences  are  displayed  during 
the  test  procedure,  depending  on  the  skill  level  of  the  user. 
Each  video  sequence  can  be  accompanied  by  additional 
information  such  as  part  numbers,  order  information,  notes 
etc.  In  addition,  any  commentaries  or  sounds  can  also  be 
included  in  the  videos. 

Both  video  players  and  optical  disks  can  be  used  as  video 
systems.  The  video  system  is  controlled  by  the  computer. 

The  video  support  system  can  be  controlled  both  directly  from 
a  test  program  and  from  a  Graphical  User  Interface  (GUI)  as  a 
user  aid.  To  this  effect,  the  video  sequences  for  a  respective 
procedure  are  selected  directly.  The  user  can  select  any 
desired  video  sequence  via  the  general  training  mode. 

Each  video  sequence  can  be  operated  via  a  GUI  of  the  Video 
Support  System  (VSS)  or  via  a  remote  control  unit  located 
directly  at  the  work  place  for  the  specimen.  A  parallel  video 
monitor  can  be  connected  to  the  video  support  system  for  this 
purpose.  This  is  useful  with  larger  test  systems  where  the  test 
specimen  is  located  a  few  meters  away  from  the  standard 
operator's  console. 

The  video  sequence  can  be  operated  as  on  a  standard  video 
player  by  using  the  GUI  or  video  remote -control  unit,  i.e.  the 
standard  functions  such  as  start,  stop,  pause,  acknowledge, 
repeat,  fast  forward  etc.  can  be  performed.  These  functions, 
however,  are  only  possible  within  the  selected  video 
sequences. 


3.6  Evaluating  Results 

3.6.1  Reports 

TICAMS  feature  two  separate  repot’  systems.  The  first  is  a 
test-system  log  book,  in  which  all  of  the  operations  can  be 
recorded.  These  operations  refer  to  the  defined  activities  such 
as  initiating  an  activity,  starting  a  test  program,  terminating  a 
test  program  etc. 

The  log  book  can  be  inspected  by  a  supervisor  and  can  be 
deleted  after  backup  copies  have  been  made. 

TICAMS  also  features  another  report  system  in  which  the 
individual  test  results  during  the  test  sequence  and  the 
associated  diagnostic  results  are  stored.  The  stored  test  results 
can  then  be  printed  out  in  hard  copy  form  or  displayed  on  the 
screen.  The  backup/restore  functions  can  be  used  to  store  the 
reports  to  tape/diskette  from  which  they  can  be  recalled  when 
necessary. 


3.6.2  Statistics 

The  report  system  for  the  test  results  can  also  be  used  to 
compile  statistics  using  the  various  test  results.  If  necessary, 
these  can  also  be  compiled  via  a  network  for  several  test 
systems.  This  enables  weak  points  in  the  specimens  and  tester 
problems  to  be  determined. 

3.6.3  Quality-Assurance  Systems 

A  further  feature  of  the  TICAMS  system  is  that  it  can  be 
connected  to  a  quality-assurance  system.  To  this  effect,  the 
test  and  diagnostic  results  are  transmitted  to  the  quality- 
assurance  system.  One  such  tool  for  this  purpose  is  the 
FRACAS  system.  The  interface  for  this  system  is  already 
implemented  in  TICAMS.  FRACAS  (Failure  Reporting 
Analysis  Corrective  Action  System)  is  used  to  locate  weak 
points  in  the  specimen  and,  after  successful  analysis, 
implements  corrective  measures  by  redesigning  the  test 
subject. 

Like  the  interfaces  for  the  electronic  documentation  systems, 
the  interface  for  the  quality-assurance  system  is  also  freely 
expandable  with  the  result  that  other  quality-assurance 
systems  can  be  connected  to  TICAMS. 

3.7  Networking 

As  shown  in  figure  4-1,  all  of  the  systems  implemented  in 
TICAMS,  such  as  the  electronic  documentation  tools  and  all 
of  the  external  tools  such  as  FRACAS,  configuration  control 
and  warehouse-management  systems  can  be  addressed  via  a 
network. 

TICAMS  functions  as  standard  with  a  LAN/Ethemet 
connection,  thus  providing  a  wide-ranging  network  facility.  A 
remote  data  transfer  unit  can  also  be  connected  for  further 
requirements  (see  section  5.3). 

This  feature  therefore  ensures  that  new  software  tools  can  be 
connected  to  TICAMS  at  any  time. 
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Additional  expansions  to  TICAMS  selected  by  the  user, 

4.  STRUCTURE  OF  TICAMS  depend  on  both  the  test  specimen  which  is  to  be  supported  by 

TICAMS  and  on  the  user  environment. 

TICAMS  is  designed  for  implementation  in  a  variety  of 

application  areas.  In  order  to  meet  the  latest  developments  in  The  complexity  of  the  test  subject  will  determine  the  need  for 

the  area  of  Integrated  Logistic  Support,  TICAMS  has  been  expert  systems  for  diagnostic  support,  configuration-control 

designed  as  a  fully  modular  system  (see  figures  4- 1  and  4-2).  modules/tools  and  with  variable  specimen  configurations,  the 

TICAMS  can  be  modified  in  accordance  with  the  hardware  of  incorporation  of  failure  analysis  systems  in  the  area  of  quality 

the  test  system.  The  system,  therefore,  is  largely  independent  assurance  or  direct  access  facilities  to  the  warehouse 
of  the  test  configuration.  The  range  of  applications  covers  management  systems, 

both  stationary  and  mobile  test  equipment  in  the  military  and 

civil  fields.  This  configuration  flexibility  is  due  to  the  The  required  software  modules  are  fully  supplied  either 

modular  hardware  and  the  modular  software  structure  of  the  directly  by  TICAMS  or  via  a  network.  In  this  way,  the  system 

test  equipment  can  be  configured  according  to  the  specific  requirements. 

In  order  to  use  the  full  capability  of  TICAMS  with  regard  to  The  user-support  system  via  on-line  documentation  requires  a 

minimum  test,  diagnosis  and  integration  time  and  maximum  high  degree  of  adaptability.  TICAMS  provides  interfaces  for 

user  support  at  optimum  cost  conditions,  the  application  test  documentation  tools  which  can  be  freely  defined  in  the 

programs  should  be  designed  as  modular  structures,  in  as  far  system.  The  system  is  easily  expanded  without  requiring 

as  this  is  possible.  The  test  language  is  not  of  prime  modification  to  existing  applications.  In  this  way,  existing 

importance.  TICAMS  is  currendy  being  used  to  generate  applications  can  be  expanded  subsequently  by  means  of  on- 

application  test  programs  in  ATLAS  -  a  language  which  is  line  documentation,  e.g.  by  a  video  support  system, 

specially  designed  for  signal  simulation,  measurement  and 
evaluation. 


On  line  Universal  Test-4  Maintenance  Maintenance  Data 

Documentation  Management  System  (UTMS)  Management 


Figure  4-1  Interface  structure  of  TICAMS 
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Figure  4-2  Modular  structure  of  TICAMS 
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5.  TICAMS  IMPLEMENTATIONS 

Figure  5- 1  shows  the  planned  fields  of  application  for 
TICAMS.  The  system  is  primarily  suited  for  implementation 
in  areas  where  complex  electronics  and  RF  signal  processing 
require  the  use  of  modem  test  equipment  with  the  associated 
comprehensive  test  documents,  test  subject  descriptions, 
training  documents  and  efficient  network  structures. 

These  applications  cover  the  entire  spectrum  of  military  test 
applications  -  both  as  stationary  and  mobile  test  stations,  in  the 
field  of  radar,  electronic  warfare  and  avionics  equipment  and 
for  digital  and  video  applications  as  well  as  microwave 
applications. 

Implementation  in  civil  applications  is  also  planned  for 
production  processes  in  the  avionics,  automobile  and  general 
RF-equipment  industries  as  well  as  in  the  field  of  quality 
assurance. 

S.l  Tornado  Seif  Protection  Jammer 

TICAMS  is  currently  being  used  by  the  German  Air  Force  for 
its  Tornado  Self  Protection  Jammer.  An  expert  system  is  first 
being  implemented  in  the  Aerospace  Ground  Equipment 
(AGE)  for  diagnosis  operations. 


In  addition,  the  video-supported  user  guidance  system  and  the 
full  electronic  documentation  system  are  also  being  used. 
CIROS,  Hyperdoc  and  VSS  have  been  implemented  as 
documentation  tools.  The  CIROS  system  supplies  the 
configuration  control,  replacement-part  catalogue  and  the 
illustrated  parts  catalogue.  Hyperdoc  is  being  used  for  the 
functional  descriptions  and  VSS  for  all  of  the  videos.  Figure 
5-2  shows  how  TICAMS  has  been  implemented  in  the 
Tornado  Self  Protection  Jammer.  The  station  covers  the 
signal  and  measurement  range  from  digital  to  microwave 
levels. 

52  Eurofighter  ECR  90  Radar 

Further  components  for  TICAMS  have  been  implemented 
within  the  framework  of  the  development  contract  for  the 
Eurofighter  ECR  90  Radar.  These  are.  in  particular,  the 
connection  to  the  FRACAS  quality-assurance  system  (Failure 
Report  Analysis  Corrective  Action  System)  and  the  expansion 
of  the  report  systems  for  performing  statistical  evaluations 
over  several  specimens  and  test  systems.  These  expansions  are 
mostly  due  to  the  ILS  requirements  as  defined  for  the 
Eurofighter  ECR  90  Radar. 
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The  test  and  diagnostic  results  are  to  be  transferred  via  RDT  to 
a  central  location  for  the  self-teaming  facility  of  the  expert 
system.  From  this  location,  the  knowledge  bases  for  the  test 
specimens  will  be  maintained  via  the  self-learning  mode  and 
the  new  knowledge  bases  will  be  loaded  into  the  test  systems 
via  RDT. 

The  external  configuration  control  and  software  maintenance 
and  modification  will  be  performed  via  a  shared  X-mode. 

This  mode  enables  a  central  location  to  participate  in  the 
execution  of  the  application  parallel  to  the  test  operator, 
thereby  increasing  the  effectiveness  and  speed  with  which 
software  problems  are  located.  Furthermore,  software  updates 
can  thus  be  copied  to  the  system  and  a  configuration  check 
performed  simultaneously. 


5.3  Growth  Potential 

The  implementation  of  further  T1CAMS  components  is 
planned  for  the  future. 

Among  other  developments,  TICAMS  is  to  be  connected  to  a 
remote  data  transfer  (RDT)  system  so  that 

the  self-leam  capability  of  the  expert  system  can 
handle  overlapping  tests  more  rapidly 

external  configuration  control  and  software 
monitoring  and  modification  can  be  performed. 


Figure  5-1  Application  areas  for  TICAMS 
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Figure  5-2  Realisation  of  TICAMS  as  TSPJ  Tester 
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6.  CONCLUSION 

In  the  future,  the  experience  gained  regarding  life  cycle  cost 
relations  in  modem,  complex  systems  (especially  in  the  field 
of  avionics)  with  their  exceptionally  unfavourable  cost¬ 
intensive  conditions  during  their  utilization  phase  will  oblige 
all  equipment  -  and  system  manufacture,  especially  under  the 
aspects  of  a  policy  of  a  lack  of  funds,  to  promote  the  transfer 
of  costs  from  the  recurring  area  into  the  non-recurring  area 
and  thus  to  invest  funds  in  an  earlier  phase. 

This  policy  will  cause  a  slight  increase  in  development  costs 
but,  at  the  same  time,  will  reduce  the  costs  in  the  utilization 
phase  considerably. 

TICAMS  will  be  an  excellent  means  of  influencing  and 
reducing  the  recurring  cost  factor. 

The  dramatic  increase  in  life  cycle  costs  in  the  production  and 
series  phases  has  led  to  the  need  for  a  system  which  reduces 
the  costs  resulting  from  the  actual  tests,  the  required  repair 
work  and  the  associated  documentation  and  training  to  a 
minimum. 

This  system  must  resolve  the  problems  of  the  individual 
activities  in  these  phases  for  each  phase  and  activity.  This 
requires  both  an  incorporation  in  the  production  system  of  the 
manufacturer  and  in  the  complete  application  system  of  the 
user. 

This  incorporation  increases  the  demands  made  on  such  a 
system  by  the  user  with  the  result  that  extensive,  cost¬ 
intensive  training  is  required.  TICAMS  places  considerable 
emphasis  on  this  aspect,  so  as  to  minimize  the  costs  involved. 


The  support  provided  by  TICAMS  ranges  from  the  use  of 
highly-developed  diagnostic-support  systems  and  developed 
user  guidance  as  well  as  the  use  of  electronic  documentation 
to  networking  with  other  systems. 

TICAMS  is  our  solution  in  the  battle  against  rising 
life  cycle  costs. 

TICAMS  enables  simple  handling  of  highly 
complex  technology. 

Maintenance  for  us  not  only  obliges  the  ILS  team  and 
especially  the  maintenance  engineer  to  prove  that  they  are 
necessary;  when  considering  the  various  groups  of  interests 
with  their  motivation  on  both  the  user  and  the  manufacturer 
side,  maintenance  is  also  a  task  which  today  concerns 
everyone  involved  in  system  development  and  utilization  for 
different  motivations  either 

in  the  technical,  in  the  tactical  or  the  economic 
field. 

This  discipline  starts  in  the  study  and  concept  phase  of  a 
product  and  extends  to  the  production  phase  via  the  definition 
and  development  phases  and  finally  ends  when  the  series 
phase  is  completed. 

This  shows  that  the  test  systems  which  are  necessary  for  a  type 
of  equipment  must  follow  the  same  rules  and  support  the 
system  over  all  of  the  phases.  Only  such  a  system  can  support 
the  equipment  in  accordance  with  the  ILS  requirements. 

TICAMS  represents  an  effective  means  of  gaining  control 
over  life  cycle  costs. 
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SUMMARY 

The  motivation  for  development  of  virtual  crew 
system  technologies  stems  from  the  growing 
complexity  of  cockpit  interfaces  and  the  realization 
that  humans  are  spatial  beings  who  are  much  better 
equipped  to  process  and  control  information  if  it 
bears  a  spatial  and  temporal  relationship  to  the  way 
the  real  world  exists  at  the  moment.  The  virtual 
cockpit  will  provide  three-dimensional  (3-D) 
spherical  awareness,  intuitive  control  interfaces,  and 
automated  assistance  to  the  pilot.  Three- 
dimensional  visual  and  auditory  information  will  be 
presented  via  the  pilot’s  helmet,  while  tactile 
information  may  be  presented  through  micro- 
stimulators  within  the  pilot’s  g’c/ve,  which  are 
activated  as  a  function  of  hand  and/or  finger 
position  within  the  cockpit  The  intent  is  to  build  a 
virtual  cockpit  that  creates  a  representation  of  the 
look  and  feel  of  the  real  world,  to  the  extent  that 
the  interaction  with  the  display  and  control  of 
information  is  as  natural  as  possible.  The  notion  of 
a  virtual  cockpit  has  been  popularized  in  the  "Super 
Cockpit"  concept.  The  Super  Cockpit  concept 
demands  a  functional  integration  of  a  broad  range  of 
advanced  control,  display  and  avionic  technologies. 
The  successful  marriage  of  the  individual 
capabilities  that  each  of  these  technologies  represent 
demands  an  understanding  and  appreciation  of  the 
requirements  for  sensing,  processing  and  displaying 
information  to  provide  the  pilot  the  advantage  of 
3-D  spatial/situational  awareness  throughout  the 
mission.  This  paper  addresses  the  multifaceted 
development  issues  associated  with  realization  of  the 
virtual  cockpit. 

1.  INTRODUCTION 

The  guiding  principle  supporting  development  and 
integration  of  virtual  cockpit  technologies  is  that  the 
best  possible  control/display  environment  for  the 
pilot  would  be  one  that  duplicates  the  essential 
organization  and  content  of  the  real  world  as  simply 


as  possible,  while  providing  natural  and  intuitive 
control  interfaces  with  the  aircraft  and  objects  in  the 
outside  world.  In  its  ultimate  form,  the  virtual 
cockpit  would  provide  a  computer  generated  world 
(see  Figure  1)  in  wNch  all  information  is  displayed 
in  3-D  visual,  auditory  and  tactile  space,  while 
communicating  an  overall  awareness  of  the  weapon 
system  state,  relative  to  everything  in  the 
environment.  Based  on  head  position  and/or  eye 
line-of-sight,  the  pilot  would  see  a  wide  field-of- 
view  (FOV)  binocular  display  providing  an 
electronic  representation  of  the  terrain.  Way  points 
and  targets  that  may  have  been  designated  by  that 
pilot  or  a  member  of  a  previous  flight  are  overlaid 
on  the  terrain.  Also  displayed  are  the  position  and 
state  of  on-board  weapons,  ground  and  airborne 
threats  and  their  status  (e.g.,  the  lethality  envelope 
for  surface-to-air  missile  sites  may  be  shown  in 
red),  a  safe  highway  in  the  sky,  and  heading 
information  on  the  horizon.  System  status 
information  may  be  windowed  in  and  windowed  out 
as  needed.  The  optical  flow  of  objects  in  the  '  .oual 
periphery  gives  natural  cues  as  to  altitude,  attitude 
and  airspeed,  without  resorting  to  the  conventional 
display  of  flight  symbology.  An  outside-in  display 
showing  a  God’s-eye-view  of  the  aircraft  and  the 
surrounding  scene  is  selectable  to  provide  enhanced 
situational  awareness,  allowing  hand  and  voice 
intera  on  with  the  displayed  elements.  A  vibro- 
tactile  stimulator  in  the  pilot’s  g.ove  (see  Figure  2) 
provides  tactile  feedback  when  the  pilot  interacts 
with  objects  in  the  virtual  visual  environment. 
Three-dimensional  sound  is  provided  through  the 
pilot’s  earphones,  allowing  localization  of  the 
re  stive  position  of  airborne  and  ground  threat 
systems,  launched  missiles,  wingmen,  and  other 
sensed  objects  (see  Figure  3).  Through  an  on-board 
processor,  dijplayed  information  is  specially 
conditioned,  based  on  tue  mission  as  planned,  the 
events  that  have  just  preceded  that  point  in  the 
mission,  an  estimate  of  the  pilot’s  intentions,  the 
sensed  array  of  objects  within  and  beyo;  J  visual 
range,  and  the  state  of  the  pilot,  both  in  terms  of 
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physiological  and  workload  capabilities.  The 
consciousness  state  of  the  pilot  is  monitored  by 
sensors  in  the  helmet  that  monitor  such  parameters 
as  blood  flow  to  the  brain,  eyelid  behavior  and  heart 
rate.  If  the  measures  indicate  the  pilot  is 
unconscious,  and  if  the  flight  parameters  indicate 
that  a  crash  is  imminent,  then,  based  on  prior  pilot 
consent,  the  aircraft  will  be  automatically 
recovered.  Rapid  reconfiguration  of  cockpit 
controls  and  displays  is  also  under  the  pilot’s 
command.  Display/control  configurations  are 
specified  in  the  Data  Transfer  Module  (DTM)  that 
the  pilot  loads  into  the  system  prior  to  start-up.  The 
DTM  also  contains  the  pilot’s  voice  print  for  voice 
control  of  selected  system  functions  (previously 
collected  under  various  g-loadings  and  stress  states), 
along  with  the  pilot’s  head-related  transfer  function 
(for  customizing  the  3-D  sound  to  the  physical 
characteristics  of  that  pilot),  premission  system 
setup  tailored  to  that  mission,  as  well  as  to  the 
preferences  and  experience  base  of  that  pilot. 
Automated  assistance  to  the  pilot,  in  the  form  of  a 
"Pilot’s  Associate"  will  behave  much  as  a  long- 
trusted  backseater,  whose  actions  the  pilot  can 
understand  and  anticipate,  and  whose  vast  array  of 
experience  and  expertise  can  be  immediately 
brought  to  bear  during  periods  of  high  pilot 
workload  or  physiological  stress. 

The  virtual  cockpit  can  provide  its  own  embedded 
training.  While  on  the  ground,  the  pilot  can 
rehearse  the  entire  mission.  Once  the  DTM  is 
loaded,  the  system  may  be  set  to  the  rehearsal 
mode,  and  thereafter  behaves  just  as  it  would  under 
actual  flight  conditions.  The  pilot  may  elect  to 
encounter  any  number  of  ground  and/or  airborne 
threats  at  random  but  logical  times  during  the 
rehearsal. 

After  the  actual  flight,  the  virtual  cockpit  becomes  a 
boon  to  the  maintenance  technician  as  well . 
Conventional  aircraft  do  not  provide  the  opportunity 
to  review  a  reported  malfunction  exactly  the  way  it 
appeared  to  the  pilot.  Instead,  the  maintenance 
technician  must  try  to  duplicate  the  failure  on  the 
ground,  an  environment  vastly  different  from  the 
airborne  one  in  which  it  occurred.  This  often  leads 
to  reports  of  "cannot  duplicate",  resulting,  at  a 
minimum,  in  much  wasted  troubleshooting  time,  in 
addition  to  frustration  by  both  the  technician  and  the 
pilot.  Using  the  virtual  cockpit,  the  technician  is 
able  to  review  the  recorded  events  exactly  as  they 
were  displayed  to  the  pilot.  The  technician  becomes 
an  astute  observer,  since  he  or  she  knows  exactly 
what  to  look  for  and  when  to  look  for  it. 

The  virtual  cockpit  may  also  provide  defense  against 
laser  threats,  since  all  the  information  needed  to  fly 
and  fight  could  be  displayed  on  the  pilot's  helmet 


visor,  without  reference  to  the  outside  world. 
Application  of  this  same  technology  to  the  control 
of  unmanned  air  vehicles  (UAVs)  is  obvious.  The 
same  flying  skills  the  pilot  used  in  the  aircraft  could 
be  employed  to  fly  an  unmanned  vehicle  from  a 
control  facility,  remote  from  the  air  vehicle  itself. 

The  genesis  of  the  display  and  control  portions  of 
the  virtual  cockpit  lies  in  the  development  of 
advanced  helmet-mounted  display  and  sight 
(HMD/S)  systems  that  are  now  gaining  operational 
exposure.  Figure  4  shows  a  conceptual 
representation  of  a  medium  field-of-view,  binocular 
HMD.  The  U.S.  Army  is  producing  the  AH-64 
(Apache)  helicopter  with  the  IHADSS  (Integrated 
Helmet  and  Display  Sight  System).  The  IHADSS  is 
a  monocular  system  having  a  single  cathode  ray 
tube  (CRT)  attached  at  the  side  of  the  helmet  and 
projecting  a  FL1R  (forward  looking  infrared)  image 
onto  a  combiner  glass,  separate  from  the  visor.  The 
Armstrong  Laboratory  is  supporting  the  advanced 
development  of  HMDs  that  allow  the  pilot  to  see  an 
image  intensified  scene  (using  third-generation 
image  intensifiers)  or  a  raster  scene  (as  from  a 
FLER  sensor),  with  or  without  flight  symbology, 
and  viewed  so  as  to  be  superimposed  on  the  outside 
world.  This  provides  for  fail  safe  operation,  since 
the  real  world  always  remains  accessible.  These 
helmets  are  being  evaluated  in  a  comprehensive 
ground  and  flight  test  program  to  demonstrate 
ejection  compatibility,  comfort  and  safety,  optical 
and  acoustic  performance,  and  operational  utility. 
Figures  5  and  6  show  front  and  side  views, 
respectively,  of  an  HMD  configuration  developed 
by  the  Armstrong  Laboratory  and  known  as 
TOPHAT.  This  design  features  a  frangible 
combiner  that  separates  from  the  rest  of  the  helmet, 
should  the  pilot  eject  from  the  aircraft. 

2.  AVIONIC  DATA  SOURCE 
REQUIREMENTS 

It  cannot  be  stated  too  strongly  that  the  fundamental 
reason  for  providing  information  to  the  pilot  is  to 
convey  an  accurate  impression  of  the  present 
situation,  relat  ve  to  the  objectives  of  the  mission. 
The  pilot  needs  to  acquire  and  maintain  spatial 
situational  awareness.  This  requires  that  he  knows 
where  he  is  relative  to  the  terrain,  waypoints, 
landmarks,  ground  and  airborne  threats,  his 
wingmen,  ground  friendly  forces,  and  the  target(s). 
He  must  use  this  information  to  assess  what  should 
be  done  during  the  next  few  moments  and  how  that 
compares  with  the  perspective  he  has  on  the  mission 
as  planned,  or  as  recently  reformulated  within  his 
mind.  Since  the  pilot  is  totally  dependent  on  the 
viability  of  his  aircraft,  it  is  crucial  that  aircraft  and 
weapon  status  information  be  provided,  but  only 
when  needed  to  convey  information  important  at  the 


moment,  or  to  construct  strategies  to  overcome  new 
contingencies. 

Many  of  the  functional  assists  to  the  pilot  in  the 
areas  of  mission  planning,  tactics  development, 
system  status  monitoring,  situation  assessment  and 
design  of  the  pilot-vehicle  interface  (especially  for 
head-down  displays)  are  being  addressed  within  the 
Pilot’s  Associate  program  being  managed  by  the 
Wright  Laboratory  at  Wright-Patterson  Air  Force 
Base,  Ohio.  It  is  not  within  the  scope  of  this  paper 
to  address  the  capabilities  being  developed  under 
that  effort,  though  they  are  relevant.  A  detailed 
discussion  of  the  Pilot’s  Associate  program  may  be 
found  in  the  paper  by  Banks  and  Lizza  (Ref  1).  See 
the  paper  by  Hopper  (Ref  2,  immediately  following 
this  report)  for  a  description  of  technologies 
supporting  development  of  large  area  head-down 
displays. 

The  virtual  cockpit  will  depend  on  inputs  from  a 
broad  range  of  avionic  sensors  for  the  collection, 
processing  and  display  of  information  in  real  time. 
For  purposes  of  discussion  here,  real  time  will  be 
considered  to  be  within  one  frame-period  of  when 
the  data  was  collected.  The  most  stringent 
requirement  has  to  do  with  above  ground  level 
(AGL)  information,  from  whatever  means  is 
available,  to  assure  the  pilot  maintains  safe  vertical 
ground  clearance.  Determination  of  horizontal 
situation  may  be  provided  by  an  Inertial  Reference 
System  (IRS),  the  Global  Positioning  System  (GPS), 
or  an  airborne  surveillance  capability.  As  discussed 
below,  this  positional  information  may  be  fused 
with  either  sensed  date  (as  from  a  FLIR,  Low  Light 
Level  TV,  or  radar)  or  an  on  board  terrain  database 
to  provide  the  pilot  scene  context  information  on  the 
HMD. 

3.  SENSOR  INTEGRATION 

Information  displayed  in  the  virtual  cockpit  must 
represent  an  integration  of  on  board  stored  digital 
terrain  elevation  and  feature  data,  with  sensed,  but 
processed,  all-aspect  air-to-air,  together  with  air-to- 
ground,  radar  inputs.  Supporting  and/or 
verification  information  may  be  provided  by  a  broad 
range  of  avionic  systems,  including  threat  receivers, 
radar  altimeter,  IRS,  GPS,  Infrared  Search  and 
Track  (IRST),  Identification,  Friend,  Foe  or  Neutral 
(IFFN),  Joint  Tactical  Information  Distribution 
System  (JTIDS),  Airborne  Warning  and  Control 
(AWACS)  or  other  sources.  Stanley  (Ref  3) 
discusses  challenges  and  issues  in  achieving  an 
integrated  sensor  suite.  Today,  issues  of  cost, 
volume  and  weight  represent  primary  technological 
hurdles  facing  developers  of  advanced  avionic 
systems.  As  advances  in  sensor  technologies  are 
made,  the  impact  of  software  capable  of  controlling 


and  managing  a  federated  sensor  suite  supporting 
the  virtual  cockpit  in  real  time  will  become  even 
more  critical.  The  software  must  be  able  to  allocate 
sensor  resources  based  on  random  mission  events, 
known  events,  and  at  times,  insufficient 
information.  This  dynamic  reconfiguration  must  be 
capable  of  fusing  on  board  and  off  board  sensor  and 
database  assets  and  be  supported  by  sufficient 
intelligence  (perhaps  via  Pilot’s  Associate 
technology)  to  be  able  to  display  a  coherent 
synthesized  graphic  output  to  the  pilot. 

4.  DATA  PROCESSING  AND  THROUGHPUT 
REQUIREMENTS 

Thus,  one  of  the  most  difficult  challenges  to  be 
dealt  with  in  the  development  of  the  virtual  cockpit 
has  to  do  with  data  fusion  across  the  suite  of 
sensors,  stored  terrain  and  intelligence  information, 
and  complementary  data  received  in-flight.  There 
are  several  aspects  of  the  data  fusion  issue  that 
warrant  discussion  here.  The  first  consideration  is 
one  of  spatial  correlation  of  data  across  the  several 
spectral  sources.  The  correlational  process  must 
remain  robust  under  variations  of  spectral  power 
(since  different  parts  of  the  spectrum  are  being 
sensed  by  different  imaging  sensors),  and  spatial 
distortion  due  to  atmospheric  and/or  optical 
attenuation  across  sensors.  Any  display  of  fused 
raw  imagery  (e.g.,  a  integration  of  FLIR  and  Low 
Light  Level  TV)  must  be  sufficiently  registered 
spatially  so  that  the  combined  images  overlaid 
throughout  the  sensed  area  make  sense  visually  to 
the  pilot  (there  will  be  more  discussion  of  this  in 
Section  5.,  below).  If  the  sensed  data  is  processed 
and  target  recognition,  or  other  rlgorithms,  are 
applied  to  translate  from  the  sensor  format  to  an 
encoded  graphical  representation  of  the  scene,  the 
correlation  problem  may  become  even  more  critical 
due  to  the  inability  of  the  processing  algorithms  to 
recognize  confusions  that  the  pilot  may  be  able  to 
resolve.  Similarly,  if  an  automatic  target  recognizer 
(ATR)  produces  false  alarms  above  some  rate  the 
pilot  finds  acceptable,  its  output  will  be  either 
disregarded,  or  considered  to  add  unreasonable 
workload  and  be  turned  off  by  the  pilot.  This  is  a 
critical  area  for  research,  since  performance  of  the 
ATR  is  likely  to  have  its  greatest  impact  during  the 
very  most  heavily  loaded  segments  of  the  mission. 

Once  the  information  is  extracted  from  sensors 
and/or  a  stored  database,  the  temporal  capabilities 
of  the  data  processing  routines  as  well  as  the 
graphics  generator  used  to  draw  the  scene  on  the 
image  source  within  the  HMD  become  the  critical 
limiting  factors  driving  the  utility  of  the  information 
to  the  pilot.  Whatever  processing  delays  are  present 
will  be  added  to  head  position  sensing  (i.e.,  helmet 
mounted  sight)  system  delays.  See  Kocian  (Ref  4) 
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for  a  discussion  of  "Advanced  Helmet  Mounted 
Display  Electronics”  and  "Magnetic  Helmet 
Mounted  Sight"  design  and  performance,  reflecting 
today's  technology  available  for  supporting  HMD 
engineering  and  human  factors  research. 

A  major  goal  of  the  software  development  efforts 
supporting  the  virtual  cockpit  technologies  must  be 
to  place  tight  restrictions  on  allowable  throughput 
delays.  Successful  development  will  demand  the 
cooperative  efforts  of  software  and  hardware 
designers  alike,  since  the  basic  architecture  of  the 
avionics  system  (e.g.,  distributed  versus  federated) 
will  place  physical  boundaries  on  data  processing 
and  distribution  speeds.  Another  important 
limitation  on  the  temporal  capabilities  of  the  system 
is  imposed  by  the  nature  of  the  databus.  The  virtual 
cockpit  will  require  several  fiber  optic  data  links 
among  the  various  sensors,  processors  and  display 
units. 

5.  HUMAN  SENSORY,  PERCEPTUAL  AND 
COGNITIVE  REQUIREMENTS 

The  primary  purpose  of  the  virtual  cockpit  is  to 
convey  situational  awareness  to  the  pilot.  The 
preceding  sections  of  this  paper  have  emphasized 
that  several  attributes  of  the  hardware  and  software 
that  drive  sensor,  processing  and  display  capabilities 
also  drive  the  pilot’s  ability  to  formulate  an  accurate 
impression  of  the  situation.  There  are  numerous 
human  factors  issues  that  must  be  addressed  and 
resolved  to  assure  that  the  pilot’s  requirements  from 
the  sensory  (i.e.,  psychophysical),  perceptual  and 
cognitive,  standpoints  are  satisfied.  Furness  (Ref  5) 
discusses  many  of  these  issues  in  the  context  of 
being  challenges  to  the  human  factors  research 
community.  Several  of  the  issues  raised  by  Furness 
are  also  challenges  to  the  avionics  development 
community  and  have  their  counterparts  in  terms  of 
the  hardware  and  software  performance  driving  the 
virtual  cockpit.  A  fundamental  question  has  to  do 
with  how  large  the  field-of-view  (FOV)  must  be  to 
provide  the  pilot  with  "adequate"  situation 
awareness.  Aside  from  the  obvious  issue  of  what  is 
meant  by  "adequate",  the  answer  to  this  drives  the 
sensor  array  size,  resolution  of  the  resulting  display, 
and  processing  speed  requirements  of  the  display 
processor.  If  the  scene  is  being  generated  by  a 
graphics  processor,  similar  resolution  and 
processing  speed  requirements  are  levied. 
Additionally,  when  the  pilot  moves  his  head  and/or 
eyes,  the  scene  must  follow  correspondingly  so  as 
not  to  produce  annoying  or  disorienting  effects. 

The  display  and  graphics  processors  must  be  able  to 
maintain  temporal  coherence  with  the  helmet- 
mounted  sight  (head  and/or  eye  position)  sensing 
system.  Related  to  the  issue  of  resolution  is  that  of 
registration  accuracy.  If  the  computer  generated 


scene  is  to  be  projected  onto  the  real,  and/or  the 
sensor  generated  scene,  all  scene  elements  must  be 
registered  properly.  Similarly,  when  symbolic 
information  is  overlaid  on  the  displayed  scene,  its 
position  and  psychophysical  attributes  must  be 
carefully  designed  to  assure  it  is  where  and  what  it 
should  be  to  best  convey  its  meaning  to  the  pilot. 

In  addition  to  conventional  human  factors 
considerations  of  symbol  meaning  and  legibility  will 
be  problems  of  obscuration  and  integration  of 
symbology  with  other  elements  in  the  scene. 

Ideally,  the  luminance  and  chromatic  contrast  of  the 
symbology  would  be  manipulated  by  the  graphics 
processor  to  maintain  symbol  visibility,  relative  to 
the  background.  These  issues,  together  with  a  great 
many  others  dealing  with  the  organization  of 
information  in  the  virtual  cockpit  and  its  meaning  to 
that  individual  pilot  can  be  expected  to  drive  the 
cognitive  utility  of  the  displayed  visual  environment. 

The  3-D  auditory  information  will  have  similar, 
though  less  stringent,  requirements  for  resolution 
and  registration  with  both  the  real  world  and  the 
visual  portion  of  the  virtual  cockpit.  One  reason  for 
this  is  the  inherently  poorer  resolution  capabilities 
of  the  auditory  localization  system  of  the  human 
(i.e.,  we  can  localize  the  direction  of  sounds  to 
within  approximately  5  to  8  degrees,  depending  on 
their  relative  location),  as  opposed  to  the  visual 
system  (i.e.,  the  ability  to  resolve  two  bars 
separated  by  1  minute  of  arc  corresponds  to  a 
"normal"  visual  acuity  under  nominal  viewing 
conditions).  Another  reason  is  that  the  auditory 
information  may  be  used  primarily  to  produce  an 
orienting  response  from  the  pilot,  to  direct  his 
attention  to  the  GENERAL  area  of  an  object  being 
displayed  visually. 

The  resolution  and  registration  of  tactile  information 
(if  provided)  will  be  dependent  on  the  position 
sensing  system  being  used  on  the  pilot’s  hand, 
together  with  the  resolution  of  the  tactile  micro¬ 
stimulators  in  the  pilot’s  glove,  and  the  registration 
of  that  information  with  the  visual  virtual 
environment.  The  development  of  micro-stimulators 
can  be  expected  to  represent  a  substantial  challenge 
to  that  community. 

6.  SYSTEM  INTEGRATION  REQUIREMENTS 

Although  the  integration  of  the  avionic  suite 
supporting  the  virtual  cockpit  has  been  briefly 
discussed  in  the  sections  above,  an  in-depth 
treatment  of  system  integration  requirements  is 
beyond  the  scope  of  this  paper.  However,  there  are 
some  cockpit  technology  integration  issues  that  must 
be  addressed  in  the  process  of  developing  the  fully 
functional  virtual  cockpit.  During  the  next  few 
decades,  and  prior  to  having  a  full  color,  wide  FOV 


virtual  cockpit  capability,  there  is  a  requirement  to 
make  the  virtual  cockpit  visual  display  compatible 
with  other  visual  displays.  A  central  concern  will 
be  the  mutual  utility  of  head-down,  head-up  and 
helmet-mounted  displays,  and  how  information  on 
one  display  medium  can  be  made  to  be  compatible 
with  (or  not  disruptive  to)  information  on  another. 
This  will  require  the  integration  of  head/eye  position 
data  with  display  processing  so  that  some  or  all  of 
the  information  may  be  blanked  on  one  display  or 
the  other,  depending  on  what  information  the  pilot 
is  looking  at,  the  mission  segment,  or  the  task  being 
performed  at  the  moment. 

As  new  avionic  capabilities  become  operationally 
viable,  their  impact  on  the  spatial  and  temporal 
information  processing  requirements  of  the  pilot 
must  be  paramount  considerations  in  how  they  are 
integrated  into  the  rest  of  the  avionic  architecture. 
Integrated  properly,  the  virtual  cockpit  and 
supporting  avionic  technologies  promise  immense 
leaps  in  pilot,  system  and  mission  capabilities  to  fly, 
fight  and  survive  throughout  all  flight  environments. 


REFERENCES 

1.  Banks,  S.B.  and  Lizza,  C.S.,  "Pilot’s  Associate: 
A  Cooperative,  Knowledge-Based  System 
Application",  IEEE  Expert,  June  1991, 

pp  18-29. 

2.  Hopper,  D.G.,  "Panoramic  Cockpit  Displays", 
in  "Advanced  Aircraft  Interfaces:  The  Machine 
Side  of  the  Man-Machine  Interface",  AGARD 
CP-521,  1992,  Paper  9. 

3.  Stanley,  J.,  "PAVE  PACE:  Concepts  for  Sensor 
Integration",  in  "Twelfth  Annual  IEEE/AESS 
Dayton  Chapter  Symposium:  Future 
Avionics... What  Direction  Will  It  Take?", 

4  December  1991,  pp  64-70. 

4.  Kocian,  D.F.,  "Visually  Coupled  Systems 
(VCS):  The  Virtual  Panoramic  Display  (VPD) 
System",  NASA  Conference  Publication  3127, 
Vol  2,  1992,  pp  548-561. 

5.  Furness,  T.A.,  "The  Super  Cockpit  and  its 
Human  Factors  Challenges",  Proceedings  of 
the  Human  Factors  Society  -  30th  Annual 
Meeting,  1986,  pp  48-52. 


rntmem-A 


Figure  2.  Vibro-Tactile  Stimulator  and  Finger  Position  Sensor  in  Pilot’s  Glove 


Figure  3.  Three-Dimensional  Audio  World 
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Figure  6.  Side  View  of  TOPHAT  Helmet  Mounted  Display 


Discussion 


QUESTION  S.  FRANCALANCI 

Will  the  development  of  a  virtual  cockpit  include  substitution  of  Head- 
Up  display  by  Helmet -Mounted  display? 

REPLY 

Yes,  but  not  immediately  due  to  some  technical  (weapon-aiming  related) 
problems.  The  future  cockpit  will  extend  the  display  area  into  the  area 
currently  used  by  HUD  optics  and  electronics. 
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1.  SUMMARY 

The  great  challenge  of  today’s  cockpit  designers  is  to 
provide  the  21st  century  pilot  the  necessary  situation 
awareness  to  be  effective  in  combat  This  situation 
awareness  is  difficult  to  achieve  because  a  pilot  has  to 
look  at  numerous  dials  and  indicators,  multiple  small 
displays  with  different  range  scales  and  ownship 
locations,  and  distorted  radar  images.  Today  the  pilot 
must  fuse  all  information  and  be  able  to  make 
immediate  critical  decisions  in  a  combat  environment 
Part  of  the  solution  to  this  information  overload  problem 
is  a  panoramic  agile-window  display.  The  advantages 
of  a  large  area  display  system  were  recently 
demonstrated  in  the  Panoramic  Cockpit  Control  and 
Display  System  research  program.  The  principal 
objective  result  was  a  45%  increase  in  pilot  combat 
effectiveness. 

The  key  to  the  implementation  of  the  panoramic  cockpit 
concept  is  a  large  area  display  together  with  a  helmet- 
mounted  equivalent  of  the  present  day  head-up  display. 
The  large  area  head  down  system  uses  direct  view  or 
projected  view  to  create  an  aggregate  display  area  of 
650-2000  cm2.  The  status  of  cockpit  displays  is 
reviewed  with  an  emphasis  on  hardware.  Several 
technologies  are  being  developed  simultaneously  and  are 
analyzed  here  against  requirements  of  our  cockpit 
visions  for  new  systems  such  as  the  F-22  and  RAH -66, 
retrofit  programs  like  the  C-130/C-141,  and  advanced 
fighter  and  transport  concepts. 

LIST  OF  SYMBOLS 
AM  active  matrix  (T,M,P,R) 

addressing  of  LC,EL,DM,MP 
AMLCD  active  matrix  (T,M,P)  LCD 

AMELD  active  matrix  (T)  ELD 

AMDMD  active  matrix  (R)  DMD  display 


AMMPD  active  matrix  (R)  MPD 

ANV1S  aviator’s  night  vision  imaging  system 

AOhl  acousto-optic  modulator 

a-Si  amorphous  silicon 

ASIC  application  specific  integrated  circuit 

ATF  advanced  tactical  fighter  (F-22) 

AW  ACS  airborne  warning  and  control  system 

BVR  beyond  visual  range 

cd  candela  (unit  of  luminous  intensity) 

CDG  cockpit  display  generator 

CGH  computer  generated  hologram 

CHDD  color  head  down  display 

CMD  cockpit  mounted  display 

CRT  cathode  ray  tube 

CVC  combat  vehicle  crew 

DMD  deformable  mirror  device 

EBS  e-beam  pumped  semiconductor 

(laser  array) 

ELD  electroluminescent  display 

EMD  engineering  &  manufacturing  development 

F-22  air  superiority  fighter 

FCRT  fiat  CRT 

footcandle.fc  1m  ft'2  (10.7639  lx) 

footlambert/L  (pi)'1  cd  ft'2  (3.42626  cd  m  2) 

FLIR  forward  looking  infrared 

FOR  field  of  regard 

FOV  field  of  view 

FPCD  flat  panel  cockpit  display 

(LED, AMLCD, etc.) 

GPS  global  positioning  system 

HDD  head  down  display 

HMD  helmet  mounted  display/sight 

HMS  helmet  mounted  sight 

HUD  head  up  display/sight 

IFOR  instantaneous  FOR 

IFOV  instantaneous  FOV 

ILO  inside  looking  out 


*  Published  in  "Advanced  Aircraft  Interfaces:  The  Machine  Side  of  the  Man-Machine  Interface",  AGARD  CP-521, 
1992,  Paper  9.  Conference  Proceedings  of  the  63rd  Avionics  Panel  Meeting/Symposium  held  in  Madrid,  Spain,  18- 
22  May  1992.  Published  by  the  NATO  Advisory  Group  for  Aerospace  Research  and  Development  (AGARD) 
Avionics  Panel  (A VP). 
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JCO 

Joint  Cockpit  Office  of  AL,WL 

JTIDS 

joint  tactical  information 
distribution  system 

lambertJL 

lm  cm'2,  (pi)'1  cd  cm'2  [obsolete] 

LC 

liquid  crystal 

LCD 

liquid  crystal  display 

LED 

light  emitting  diode  (display) 

LH 

RAH-66 

lumen,  lm 

cd  sr  (unit  of  luminous  flux,  F) 

lux,  lx 

lm  m'2  (unit  of  illuminance,  E) 

M 

magnetic  (NiFe) 

MAWDS 

multifunctional  agile  window 
display  system 

MFD 

multifunction  display 

MPD 

microtip-field-emission-plate 
phosphor  display 

MTAC 

mission/task  adaptive  cockpit 

NVG 

night  vision  goggle 

OLI 

outside  looking  in 

P 

plasma 

p-Si 

polycrystalline  silicon 

PALC 

plasma  addressed  LCD 

PCCADS 

panoramic  cockpit  control  and 
display  system 

PC 

panoramic  cockpit 

PCD 

panoramic  cockpit  display 

PDLC 

polymer  dispersed  liquid  crystal 

HIFOV 

high-resolution  IFOV 

pixel 

picture  element  (2-D  display) 

R 

random  access  memory  (RAM) 

RAH-66 

reconnaissance/attack  light 
helicopter,  "Commanche" 

RAMTIP 

reliability  and  maintainability 
technology  insertion  program 

RGB 

red-green-blue 

SC 

super  cockpit 

SFD 

small  format  display 

SIPE 

soldier’s  integrated  protective  ensemble 

SLM 

spatial  light  modulator 

smafhired 

small  format  high  resolution  display 

SOF 

special  operational  forces 

SSLD 

solid  state  laser  display 

sygvrew 

synthetically  generated  view  of  the 
real  world 

T 

transistor 

TFT 

thin  film  transistor  (in  Si,  CdSe) 

TN 

twisted  neumatic  (LC) 

VDMS 

visual  display  modelling  system 

VFR 

visual  flight  rules 

voxel 

volume  element  (3-D  display) 

WL 

Wright  Laboratory 

YGB 

yellow-green-blue 

XGA 

extented  graphics  array 

x-Si 

single  crystal  silicon 

2.  INTRODUCTION 

This  paper  addresses  hardware  developments  needed  to 
create  panoramic  cockpit  display  systems. 

2.1  Threat 

Present  day  combat  pilots  suffer  from  information 
overload.  The  result  is  decrease  or  loss  of  situational 
awareness  (SA)  at  times  when  it  is  most  important 
Beyond  visual  range  (BVR)  objects  are  also  difficult  to 
envision  and  fit  into  a  total  picture.  These  threats, 
together  with  night,  in-weather,  low-level  flight 
conditions  and  the  laser  threat,  may  require  panoramic 
displays  in  21st  century  cockpits.  The  future  military 
pilot  might  then  have  to  fly  some  mission 
segments-notably  ingress,  attack,  and  egress-without 
any  benefit  whatsoever  of  looking  out  of  the  cockpit 
The  direct  world  view  now  acquired  by  the  pilot’s 
unaided  eyes  might  be  denied  even  during  VFR  flight 
conditions.  Fortunately,  there  is  an  opportunity, 
technologically  speaking,  to  provide  a  degree  of 
information  that  is  not  only  sufficient,  but  which  may 
significantly  surpass  in  many  regards  that  currently 
available  through  the  canopy  on  a  clear  weather  day. 
The  canopy  would  be  closed  by  curtains  or  by  an 
electrically  controlled  opaquing  layer.  Then  an  artificial 
world  view,  called  virtual  reality  by  some,  would  be 
created.  The  control  and  display  system  might 
logically  evolve  as  an  extension  of  present  day  night/in¬ 
weather  instrumented  flight  systems.  Eventually,  a  large 
area  display  suite  may  become  necessary  to  survival,  let 
alone  mission  success.  Such  a  system  might  be  a  2000 
cm2  (300  in2)  head-down  color  multi-function,  multi¬ 
window  display  coupled  with  a  head-mounted  system 
and  would  have  to  operate  with  a  clear  as  well  as  an 
opaqued  canopy. 

22  PCCADS 

2.2.1  Results 

A  program  of  studies  conducted  by  this  office  has 
demonstrated  an  effective  way  to  deal  with  the  threat 
just  described.  The  approach  in  this  program,  entitled 
"Panoramic  Cockpit  Control  and  Display  System 
(PCCADS),"  is  to  provide  the  pilot  with  large  area 
displays  and  a  helmet-mounted  off-axis  target- 
acquisition  weapon-targeting  system.  There  were  two 
projects,  one  focused  near  term,  the  other,  far  term.1 

The  PCCADS  2000  cockpit  was  designed  to  be 
realizable  with  1993  technology  and  featured  a  25  cm 
(10  in.)  square  tactical  situation  display  and  two  13  cm 
(6  in.)  square  secondary  multifunction  displays  on  either 
side.2  All  displays  were  full  color  capable  with  a  total 
area  of  1 1 10  cm2  (172  in2).  The  test  mission  was  for  an 
F-15E.  A  28%  increase  in  exchange  ratio  was  achieved 
versus  the  standard  F-15E  cockpit  Coupling  this  large 
display  with  a  helmet  mounted  display  (HMD)  for  off 


axis  target  acquisition  resulted  in  a  45%  increase.  The 
PCCADS  2000  concept  is  illustrated  in  Figure  1.  The 
F-15E  displays  are  depicted  in  Figure  2  for  comparison 
to  die  PCCADS  2000  displays,  shown  in  Figure  3. 

The  PCCADS  2001  cockpit  is  a  variation  of  the 
PCCADS  2000  cockpit  PCCADS  2001  features  two  25 
cm  and  one  10  cm  square,  color  displays  providing  a 
total  area  of  about  1350  cm2  (210  in1).  Each  large  MFD 
can  contain  four  scalable  windows.  The  two  MFDs  are 
side  by  side  and  can  be  driven  together  as  one  large 
albeit  segmented  55  x  25  cm  display  array  with  a  5  cm 
wide  pair  of  bezel  button  bars  bisecting  it 

The  PCCADS  cockpit  was  designed  to  be  realizable 
with  beyond  2000  technology  and  featured  a  2000  cm1 
(300  in1)  display.  This  research  demonstrated  the 
payoff  in  increased  situational  awareness  from 
integrating  all  information  and  displaying  it  to  the  pilot 
on  one  large  display  format  with  secondary  information 
displayed  as  needed  in  windows.  This  effort  actually 
preceded  the  PCCADS  2000  effort,  which  was  to  show 
how  much  of  PCCADS  could  be  realized  with  today’s 
technology.  The  PCCADS  2000  cm1  test  cockpit  for 
improved  man-machine  interface  is  illustrated  in  Fig.  4. 

2.2.2  Implementation 

The  key  to  the  implementation  of  the  panoramic  cockpit 
concept  is  a  large  area  Head  Down  Display  (HDD)3 
together  with  a  helmet-mounted  display  and  sight  having 
the  capacity,  initially,  of  the  present  day  head-up  display 
(HUD)  augmented  with  off-axis  capability.  There  are 
two  complementary  approaches:  outside-looking-in 
(OLI)  and  inside-looking-out  (ILO).  In  the  OLI 
approach,  the  viewer  perceives  himself  to  be  located 
outside  looking  in  on  the  world  presented  on  the 
display.  In  the  ILO  approach,  the  viewer  perceives 
himself  to  be  located  inside  the  displayed  world  looking 
out  at  it;  a  large,  120  x  6(F,  field  of  view  is  required  for 
one  to  "think"  one  is  actually  immersed  in  the  world 
presented  on  the  dispiay(s).  The  ILO  approach  is 
achieved  today  via  the  tranparent  canopy:  the  pilot  is 
centered  in  a  real  world  with  all  display  elements 
coming  from  real  world  phenomena.  Cockpit  mounted 
display  (CMD)  technology,  represented  by  today’s 
HDDs,  represents  the  OLI  approach.  Significant 
development  is  required  for  the  CMD  technology  to  be 
used  to  implement  either  the  large  area  OLI  or, 
eventually,  the  ILO  approach.  The  head-mounted 
version  of  the  either  the  OLI  or  the  ELO  approach 
requires  a  much  more  complicated  HMD  than  one 
which  merely  transfers  sighting  or  HUD  functions  to  the 
helmet:  an  artificially  generated  view  of  the  real  world 
is  created  and  presented  on  what  we  here  term  a  "small 
format  high  resolution  display"  (smafhired).  Research 
on  helmet-mounted  virtual  cockpits  for  2020  is 


described  in  the  immediately  preceding  paper  by 
Martin.4  Research  on  smafhireds  is  included  here. 

The  OLI  approach  to  the  large  area  display  system  uses 
direct  view  or  projected  view  to  create  a  display  area  of 
some  625-2000  cm1.  An  active  matrix  liquid  crystal 
display  (AMLCD)  might  be  used  as  the  basis  of  either 
direct  or  projected  viewing:  tiling  of  smaller  displays 
(e.g.  three  10  x  20  cm  to  create  a  large,  30  x  20  cm 
display;  or  projection  from  small,  5  cm,  AMLCDs  to 
create  a  60  x  30  cm  display. 

Research  directed  at  the  creation  of  display  technology 
required  to  support  both  the  OLI  and  ILO  approaches 
and,  specifically,  to  support  a  PCCADS -capable  cockpit 
design,  including  the  simulator  dome  display  re¬ 
engineered  for  implementation  inside  of  a  cockpit-scale 
capsule,  is  reviewed.  Different  parts  of  the  display 
system  will  employ  small,  medium  and  large  area 
technologies.  In  most  cases,  the  military  technology  in 
each  regime  must,  perforce,  build  on  the  commercial. 
Availability,  one  must  recall,  comprises  usually  not  only 
the  invention  of  a  display  technology  but  also  several 
years  of  militarization  and  manufacturing  process 
development 

23  Vision 

The  Joint  Cockpit  Office  has  developed  technology 
visions  for  1996,  2000,  and  2020.  These  visions  are 
depicted  in  Figures  5,  6,  and  7. 

Today’s  modem  cockpit  design  is,  essentially,  the  F-22 
for  a  fighter,  RAH-66  Commanche  for  a  helicopter,  and 
the  C-130  RAMTIP  for  a  transport  or  special 
operational  forces  (SOF)  aircraft  RAMTIP  is  the 
Reliability  and  Maintainability  Technology  Insertion 
Program.  Each  pilot  has  650-1300  cm1  (100-200  in1) 
comprising  2-6  color  multifunction  displays  (MFD),  of 
which  any  one  of  2-3  can  be  designated  as  the  primary 
flight  instrument  and  the  rest  provide  other  mission  or 
subsystem  information  as  selected. 

The  1996  cockpit  vision  includes  a  650  cm1  (100  in1) 
full  color  flat  panel  tactical  situation  display  and  other 
fiat  panel  displays,  a  limited  Pilot’s  Associate  (PA) 
system,  and  an  all  aspect  HMD  with  3-D  audio  displays. 
This  concept  is  the  "Mission/Task  Adaptive  Cockpit 
(MTAQ”. 

The  vision  for  2000  is  a  2000  cm1  (300  in1)  panoramic 
HDD  coupled  with  a  simple  helmet  control  and 
targeting  display.  The  HUD  is  absent  in  the  2000 
vision  as  the  head  mounted  head-up  display  and  sight 
(HMDS)  system  completes  its  takeover  of  HUD 
functions,  but  in  a  vastly  superior  way  due  to  its  off- 
boresight  acquisition/designation  capability.  The  cockpit 


canopy  may  be  turned  opaque  to  optical  wavelengths9 
for  mission  segments  flown  within  the  threat  envelopes 
from  air,  ground,  or  space-bom  visual  anti-personnel 
laser  weapons.  A  graphical  world  view  is  created  from 
on-board  digital  data  bases,  the  on-board  sensor  suite, 
and  the  off-board  sensor  suite.  We  here  introduce  the 
term  synthetically  generated  view  of  the  real  world,  or 
"sygvrew",  for  die  imagery  presented  in  the  pilot's  field 
of  regard  (FOR).  This  is  to  be  contrasted  to  a  view  of 
an  imaginary,  or  fictituous,  world.  The  sygvrew  for  this 
2000  system  is  of  the  OLi  type.  Data  bases  include 
pilot  preferences,  terrain,  intelligence,  and  weather.  On¬ 
board  sensors  include  both  passive  (FLIR,  GPS)  and 
active  (radar,  lidar).  Off-board  sensors  provide  data 
linked  information  from  other  ships  in  the  flight,  JTIDS, 
AWACS,  JSTARS,  et  cetera.  This  concept  is  the 
"Panoramic  Cockpit  (PC)",  characterized  by  mature 
associate  technology,  wide  FOV/color  HMD,  control  by 
head/eye/voice.  Pave  Pace  display  generator,  and 
cockpit-wide  displays. 

Visions  beyond  2000  switch  from  the  OLi  to  the  ILO 
approach.  One  develops  the  sense  that  one  is  inside  an 
artifically  generated  scene,  or  world,  when  the  IFOV 
exceeds  about  100°  x  50°. 

The  2020  vision  is  an  encapsulated  cockpit.  The  pilot 
may  have  no  windows.  His  cabin  may  be  a 
self-contained  spheroid  embedded  within  the  aircraft  or, 
possibly,  elsewhere.  The  sygvrew  presented  in  the 
pilot's  field  of  regard  (FOR)  in  this  2020  ILO  system  is 
much  expanded  over  the  2000  OLI  system.  This 
display  system  might  be  that  of  a  simulator-only  better 
projectors  Fill  the  inside  of  the  sphere  with  a  sygvrew 
centered  on  ownship.  Alternatively,  the  sygvrew  may 
be  generated  by  a  smafhired  ("small  format  high 
resolution  display"),  or  by  a  combination  of  display 
technologies.  The  sygvrew  will  be  color  and  high 
resolution,  perhaps  in  all  parts  of  the  sygvrew  or  just  in 
the  high  resolution  part  of  the  IFOV  (HIFOV).  The 
pilot  has  the  option  of  leaving,  to  some  degree,  or  of 
removing  entirely  the  real  world  visual  effects  of 
weather  and  night  The  direct  view  display  or  helmet 
display  may  be  designated  the  primary  flight  instrument 
with  the  other  as  backup.  This  concept  is  the  "Super 
Cockpit  (SC)"  concept  characterized  by  pilot-state 
monitoring,  machine  learning,  and  3-D  virtual  world 
(visual/audio/tactile)  displays.  The  2020  vision  includes 
what  some  call  virtual  reality  coupled  with  actual  views 
from  not  only  ownship,  but  also  from  a  variety  of  othir 
platforms  via  cameras,  data  bases,  and  data  links.  Two 
concepts  for  beyond  2000  are  shown  in  Figures  6  and  7. 

These  visions  incorporate  an  evolution  to  all  aspects  of 
the  Super  Cockpit,  which  augments  present  cockpit 
technologies  with  such  things  as  graphics-generated 


views  of  the  world,  intelligent  crew  aiding,  machine 
learning,  pilot  state  monitoring,  three  dimensional 
displays,  helmet  mounted  multifunction  displays,  and 
panoramic  panel  mounted  displays. 

2.4  Goal 

Our  goal  is  to  fill  the  cockpit  design  requirement  for  a 
panoramic  display.  The  opportunity  to  do  so  is  arising 
from  research  and  development  conducted  in  our 
laboratory.  Our  strategy  is  to  pursue  multiple 
technological  approaches:  groupings  (arrays  or  seamless 
tiling  of  flat  panel  displays)  and  projectors.  We  are 
funding,  together  with  the  Defense  Advanced  Research 
Projects  Agency  (DARPA)  several  different  methods 
within  each  of  these  approaches.  The  engineering  design 
opportunity  will  be  realized  or.  the  flight  line  only  if  the 
pilot  community  accepts  our  vision.  We  expect  that  it 
will  because  of  the  environment  in  which  future  pilots 
are  now  growing  up:  panoramic  video  games  and 
learning  systems.  They  not  only  ri”  wo^nt  panoramic 
HDD  and  HMD  displays  in  the  cockpit,  they  will  expect 
and  demand  them.  Our  vision  for  the  evolution  of 
displays  and  display  generators  is  shown  in  Figure  8. 

2.5  Objectives 

These  displays  must  be  readable  in  a  variety  of  military 
situations.  Examples  include  a  pilot  at  20000  feet  on 
any  day  or  on  the  desert/forest  floor  in  a  clear  day,  a 
tank  commander,  a  field  unit  with  a  netted  portable 
computer.  The  display  must  be  suitable  to  all  other 
light  viewing  conditions  too,  including  the  simultaneous 
use  of  night  vision  devices.  Some  specifics  follow. 

2.5.1  Large  area  with  high  resolution 

Display  modules  must  measure  at  least  25  cm  (10  in) 
diagonal  and  have  color  pixel  densities  of  at  least  32 
cm'1  (80  in'1);  or  monochrome,  64  cm'1  (160  in'1). 
Several  modules  or  sizes  can  be  grouped  together  as 
necessary  to  achieve  the  total  aggregate  display  area 
required. 

2.5.2  Sunlight  readable 

Persons  with  normal  vision  must  be  able  to  read  the 
display  in  both  direct  and  occulting  sunlight.  In  each 
case  the  sun  is  not  attenuated.  Direct  sunlight  means 
the  sun  shines  on  the  display;  occulting,  into  the 
viewer’s  eye.  The  goal  inherent  in  this  requirement  is 
usually  expressed  in  terms  of  the  luminance  (light 
intensity)  emitted  and  contrast  maintained  by  the  display 
for  a  specified  illumination  condition.  Legibility  must 
be  established  under  full  daylight.  Full  daylight  for  an 
open  canopy  fighter  cockpit  is  taken  to  be  an 
illuminance  of  either  (1)  108000  lx  (10000  fc)  directly 
incident  on  the  display  with  luminance  of  1710  cd  m J 
(500  fL)  incident  at  the  specular  angle  with  respect  to 
the  test  viewing  angle,  or  (2)  21500  lx  (2000  fc) 


illuminance,  with  6850  cd  m'2  (2000  fL)  luminance  at 
specular.  Full  daylight  for  a  partially  closed  transport 
cockit  is  taken  to  be  the  same,  except  that  illuminance 
in  the  first  case  is  taken  to  be  86100  lx  (8000  fc).  A 
contrast  ratio  must  be  at  least  4.66:1  (5  grayshades) 
under  the  highest  luminance  condition  and  10:1  under 
40  %  of  the  highest;  the  goal  is  50:1  for  all  cases. 
These  requirements  translate,  for  example,  to  a  display 
white  field  luminance  in  excess  of  510  cd  m'2  (150  fL); 
for  video  applications  at  least  750  cd  m'2  (220  fL)  is 
preferred  so  that  8  green  monochromatic  grayshades  can 
be  discerned.6 

25.3  Variable  brightness 

Viewers  must  be  able  to  adjust  the  brightness  to  be 
viewable  in  a  continuum  of  over  six  orders  of 
magnitude  of  ambient  illuminance  from  108000  lx 
down  to  0.11  lx  (10000  fc  to  0.01  fc). 

15.4  Full  color  with  gray  scale 

Full  color  (8  frequencies  spanning  380-760  nm)  with  32 
gray  levels/shades  per  color  is  needed.  Color  pixels  are 
quads  comprising  four  square  subpixels  in  an  RG/GB  or 
RB/BG  scheme,  triads  comprising  three  square  subpixels 
in  an  RGB  scheme,  or  quads  comprising  four  stripe 
subpixels  in  an  YGBG  scheme.  The  quad-square 
scheme  with  an  NVG  filter  is  preferred  by  the  US  Air 
Force  and  Navy;  the  quad-stripe  without  an  NVG  filter, 
the  Army.  The  column  drivers  would  need  be  capable 
of  accepting  only  5-bits  per  addressable  pixel 
(monochrome  pixel  or  color  subpixel)  and  putting  out 
32  distinct  voltage  levels.  Some  are  suggesting  that  a 
standard  of  64-256  gray  levels  be  adopted  for  the  driver 
chips  for  flat  panel  displays.  The  higher  number  of 
levels  than  32  are  motivated  by  electronics 
considerations  (64  would  permit  the  use  of  standard,  8- 
bit  bytes)  and  sensor  considerations  (256  might  permit 
better  display  of  FLIR  images). 

255  Night  vision  compatible 
No  significant  intensity  may  be  emitted  from  the  display 
into  the  FOV  of  the  night  vision  sensor.  Night  vision 
goggles  (NVGs)  integrate  the  sensor  into  the  HMD, 
which  means  that  HDDs  and  HUDs  will  blind  the 
viewer  if  they  emit  significant  light  energy  in  the  NVG 
band.  Generation  three  (Genlll)  NVGs  amplify  600-900 
nm  (Class  A)  and  can  be  used  in  starlight  and  overcast 
conditions.  This  requirement  typically  translates  to  a 
special,  additional  filter  in  a  full  color  AMLCD  display. 
One  must  sacrifice  some  night  sky  intensity  (10%  star, 
25%  moon)  to  retain  red  (about  630  nm)  in  the  display; 
this  is  accomplished  with  two  filters:  (1)  one  in  the 
display  to  limit  its  emission  spectrum  to  wavelengths 
less  than  about  650  nm  and  (2)  a  665  nm  "minus  blue” 
cut-off  filter  in  the  NVG  to  limit  its  acceptance 
spectrum  to  665-900  nm  (Class  B).7  Alternatively,  one 


can  give  up  red  entirely  to  completely  maximize  the  use 
of  the  available  night  sky  illumination  in  the  red  and 
infrared;  one  then  has  a  yellow-green-blue  (YGB)  three 
color  display  rather  than  a  full  color,  red-green-blue 
(RGB)  display.  The  ultimate  solution  is  to  achieve 
night  vision  without  NVGs:  an  imaging  infrared  camera 
is  slaved  to  the  instantaneous  pupil  direction  and  its 
image  relayed  through  a  video  memory/processor  to  an 
HDD  or  HMD.  The  trade-off  is,  thus,  avoided  and  one 
retains  both  full  color  HDD  capability  and  complete  use 
of  available  night  sky  illumination. 

25.6  Environmental  and  Other 

Military  environmenal  and  other  specifications  must  be 

demonstrated  by  testing.* 

3.  TECHNOLOGICAL  APPROACHES 
Technological  approaches  to  large  area,  or  panoramic, 
cockpit  displays  include  array,  tiling,  and  projection. 
Arrays  are  known  commercially  by  such  names  as  video 
wall.  Tiling  retains  several  individual  displays,  as  in  an 
array,  but  removes  the  spaces  between  them.  Tiling  can 
be  accomplished  by  several  methods:  butt-coupling; 
circuit  pasting;  optical  stitching. 

3.1.  Array 

The  "glass  cockpit"  was  first  made  possible  by  CRTs. 
We  are  now  upgrading  to  more  capable  electronic 
displays  based  on  flat  panel  technology.  An  array,  or 
grouping,  of  electronic  displays  based  on  current 
technology  can  produce  a  cockpit  instrument  system 
which  is  a  close  approximation  in  flight  hardware  of  the 
PCCADS  2000  concept 

3.1.1  Cathode  tubes 

Military  avionic  CRTs  have  problems  with  reliability, 
availability,  sunlight  readability,  and  scalability.  Also, 
CRTs  cannot  be  scaled  to  2000  cm2  with  space,  weight, 
and  power  supportable  in  an  aircraft  Research  in  areas 
like  flat-neck  CRTs  may  provide  new  options,  however. 

3.1.2  Flat  Panel 

AMLCD  is  the  only  flat  panel  display  technology 
currently  capable  of  high  brightness  (sunlight  readable) 
and  full  color.  These  displays  are  under  development 
for  new  systems  and  retrofit  programs.  The  F-22  HDDs 
comprise  some  six  a-Si  AMLCDs  of  dimension  20  x  20, 
16  x  16,  10  x  8  cm  (8  x  8,  6  x  6,  4  x  3  in.)  with  200 
cm'1  (80  in'1)  color  pixel  density,  and  with  a  total  area 
of  some  1300  cm2  (200  in2).*  The  RAH-66  HDDs 
comprise  two  each  20  x  15,  10  x  10  cm  (8  x  6,  4  x  4 
in.)  a-Si  AMLCDs  for  a  total  of  some  825  cm2  (128  in2) 
in  each  cockpit  (fore  and  aft).10  The  C-13(VC-141 
retrofit  program  will  install  several  15  x  20  cm  (6  x  8 
in.)  AMLCDs  having  color  pixel  density  200  cm 1  in 
each  transport  cockpit;  a  Reliability  and  Maintainability 
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Technology  Insertion  Program  (RAMTIP)  is 
underway11  in  which  a  complete  cockpit  retrofit  of  a 
C-130  aircraft  has  been  accomplished  with  some  six  IS 
x  20  cm  (6  x  8  in.)  a-Si  and  CdSe  AMLCDs  providing 
775  cm2  (120  in1)  per  pilot  and  310  cm1  (48  in1)  for  the 
navigator.11  The  avionic  AMLCDs  for  the  two  new 
systems  and  for  other  programs  are  being  manufactured 
by  Optical  Imaging  Systems  in  Troy,  MI.13  Hosiden 
in  Japan  and  Sextant  in  Fiance  have  also  fabricated  a-Si 
AMLCDs  for  various  military  avionic  research 
programs.  Rockwell  Collins  has  demonstrated  a  video 
rate  20  x  15  cm  (8  x  6  in.)  color  AMLCD  driven  by  a 
third-generation  video  generator  and  software  that 
allows  a  range  of  100  gray  shades.14  Texas 
Instruments  has  developed  military  versions  of  its  LCD 
drivers  to  operate  displays  in  aircraft,  armored  vehicles, 
field  computers,  or  targeting  systems.13  Thus, 
AMLCDs  can  support  forward  looking  infrared  (FLER) 
and  television  video  with  motion  including  graphics. 
Schematics  of  the  F-22  and  RAH-66  EMD  cockpits  are 
provided  in  Figures  9  and  10.  Figure  11  is  a 
photograph  of  the  C-130  RAMTIP  cockpit;  Figure  12  is 
a  photograph  of  one  of  the  Litton  displays  used  in 
RAMTIP  with  the  drivers  and  connectors  shown. 

Thompson14  at  Xerox  PARC  in  Palo  Alto,  CA  has 
established  a  prototyping  facility  which  is  now 
developing  a  robust,  high  definition  AMLCD  technology 
with  a-Si  and  p-Si  TFTs.17  Xerox  plans  to  design  and 
fabricate  high  resolution  AMLCDs  having  large  display 
area  using  a-Si  and  to  research  large  area  substrates  36 
x  33  cm  (14  x  13  in.)  using  p-Si  semiconductor 
material.  Thompson  presently  sees  packaging  as  the 
main  problem,  with  multichip  designs  of  chip-on-glass 
(COG)  and  tape  automated  bonding  (TAB)  interconnects 
now,  and  integrated  drivers,  later.  A  universal 
controller  for  AMLCDs  will  be  fabricated  in  1992. 
Thompson  is  now  scaling  up  the  p-Si  woik  from  small 
area  15  cm  (6  in.)  square  to  large  area  36  cm  (14  in.) 
square.  Xerox  works  with  Standish  Industries  Inc  in 
Lake  Mills,  WI  for  LCD  processing. 

Litton  in  Canada  is  researching  AMLCDs  using  CdSe 
instead  of  Si  for  the  semiconductor  layer. 

Research  on  other  technologies  may  eventually  lead  to 
additional  flat  panel  options.  The  problem  has  been 
primarily  with  the  generation  of  the  blue  required  for 
full  color  displays.  True-blue  LEDs  which  are  bright 
and  efficient  should  be  available  for  the  display  designer 
in  the  near  future.1*  Stanley  Electric  Co.  has  an  470 
nm  LED  (using  silicon  carbide  as  the  light  emitting 
material)  on  the  Japanese  market,  and  Sharp  Corp.  will 
introduce  a  blue  LED  when  luminance  increases  by  an 
order  of  magnitude.  Some  130  mcd  at  an  efficiency  of 
0.8%  has  been  achieved  at  Nagoya  University  and  will 


be  commercialized  by  Toyota  Gosei  Co.  Efficient  red, 
yellow,  and  green  LEDs  are  widely  available,  largely  as 
a  result  of  using  GaAlAs  material.1’  A  full  color 
electroluminescent  display  (ELD)  has  recently  been 
developed  by  Planar  of  Beaverton,  OR;  again,  blue  has 
been  the  problem  with  32  lm  (3fL)  at  60  Hz  having 
been  demonstrated.10  The  display  is  presently  bright 
enough  to  be  read  in  dim  ambient  light  Neither  LEDs 
nor  ELDs  will  be  available  as  a  full-color  sunlight- 
readable  display  technology  for  many  years. 

An  active  matrix  microtip  phosphor  display  (AMMPD) 
is  another  possibility.  Here  matrix  addressing  of  the 
microtips  controls  the  intensity  of  electron  excitation  of 
pixels  comprising  phosphor  dots  or  squares. 

3 2.  Tiling 

The  full  PCCADS  concept  of  a  seamless  PCD  of  2000 
cm1  requires  a  significant  expansion  of  the  current  state 
of  the  art  in  display  technology.  One  approach  is  to 
move  the  current  discrete  displays  so  close  together  that 
one  perceives  one  large  display  rather  than  several 
discrete  multifunction  displays.  In  this  way  the  same 
windows  as  now  can  be  presented  and  it  becomes 
possible  to  present  a  seamless  panoramic  display  across 
the  tiled  array  yet  retaining  redundancy.  The  state  of 
the  art  is  represented  by  the  6144  x  2048  pixel,  152  x 
51  cm  (60  x  20  in.)  prototype  built  from  three  2K  x  2K 
CRTs  by  Masuishi,  Small,  and  MacNeil21  at  MIT  in 
Cambridge,  MA,  who  have  introduced  a  virtual 
framebuffer  architecture  to  enable  implementation  of 
two  kinds  of  seamlessness  in  tiled  systems  (CRTs  are 
used  here  for  research  purposes). 

3.2.1  Butt-coupling 

Stewart21  at  the  David  Samoff  Reseach  Center  (DSRC) 
in  Princeton  NJ  is  working  on  a  program  for  which  the 
original  goal  was  to  develop  a  30  x  20  cm  (12  x  8  in.) 
display  having  1200  x  800  color  pixels  (960000  pixels 
@  100/in.)  comprising  three  10  x  20  cm  (4  x  8  in.)  p-Si 
AMLCD  panels.  DSRC  is  now  working  with  Standish 
to  complete  a  video  rate  demonstrator  using  a  single 
panel  in  the  portrait  20  x  10  cm  (8  x  4  in.)  orientation. 
DSRC  has  done  the  design,  is  doing  the  electronics,  and 
will  produce  the  active  and  passive  glass;  Standish  will 
do  the  fill  and  seal.23  DSRC  will  then  integrate  and 
deliver  the  demonstrator  to  the  Wright  Laboratory.  This 
panel  features  integrated  drivers  and  field  sequential  full 
color.  The  integral  drivers  ("self-scanning”  display) 
reduce  the  external  lead  count  from  2400  to  46  per 
panel.  This  butt-coupling  approach  to  create  a  large 
area  display  is  illustrated  in  Figure  13. 

Firester24  at  DSRC  is  starting  design  of  a  large  area  36 
cm  (14  in.)  diagonal,  self-scanned  XGA  color  display  in 
p-Si.  The  XGA  video  standard  comprises  1024  x  768 


pixels,  each  capable  of  256  colors  non-interlaced.  In 
addition,  DSRC  will  be  designing  a  256  grayshade  data 
line  driver  (40  MHz,  20  V)  for  a-Si  XGA  LCDs  to  be 
fabricated  as  a  standard  cell  in  a  library  for  an 
application  specific  integrated  circuit  (ASIC). 

Scott24  at  Magnascreen  in  Pittsburgh,  PA  is  using  a 
different  tiling  approach.  The  research  plan  calls  for  a 
demonstration  in  the  form  of  an  8  x  6  array  of  13  x  13 
cm  (5  x  5  in.)  passive  matrix  twisted  nematic  (TN) 
liquid  crystal  panels.  The  viewer  will  perceive  the 
mosaic  to  be  a  single  display  measuring  104  x  78  cm, 
or  130  cm  diagonal  (41  x  31  in.,  or  51  in.  diagonal)  and 
consisting  of  a  total  of  640  x  480  color  pixels  (16 
colors).  If  the  Magnascreen  approach  works,  it  can  be 
used  to  tile  other  types  of  panels,  such  as  AMLCDs. 

3.2.2  Circuit  pasting 

Pasted  circuit  arrays  can  be  created  by  placing  several 
individually  fabricated  circuits  together  on  a  large 
substrate.  The  large,  pasted  circuit  is  then  to  be  driven 
as  if  fabricated  at  one  time.  The  objective  is  to  use  the 
high  temperature  processes  necessary  to  produc  circuits 
in  p-Si  and  x-Si  (silicon  substrate)  followed  by  transfer 
to  the  glass  substrate  for  display  assembly.  Several 
AMLCD  circuit  transfer  demonstrations  have  been 
accomplished  by  Kopin  in  Taunton  MA  using  DSRC 
Color  Matrix  Display  panels  (p-Si  and  x-Si,  5x5  cm, 
192x192  monochrome  pixel)  with  assembly  by  Standish. 
Such  an  x-Si  panel  has  been  successfully  operated. 
Pasting  several  of  these  smaller  displays  together  into  a 
large  display  may  be  attempted  next.26 

323  Optical  stitching 

Stitching  involves  optical  means  to  make  the  physical 
display  structure  comprising  the  discrete  pixels  and 
individual  displays  appear  to  be  one  large  display  by 
optical  interconnection  schemes. 

Macro-coupling  makes  a  segmented  display  array  appear 
to  be  one  large,  unsegmented  display.  One  means  of 
accomplishing  this  which  has  been  tried  involves  optical 
fibres,  often  with  one  fibre  per  pixel.  Fibre-coupled 
displays  have  not  demonstrated  a  visually  acceptable 
seam.  Abileah  and  Yaniv27  have  tested  a  method 
involving  a  diverging  glass  fiber-optic  faceplate  (plastic 
and  micro-channel  (hollow  pipes)  faceplates  might  be 
used  instead);  any  number  of  displays  may  be  tiled,  but 
there  is  a  20%  loss  in  resolution. 

Micro-coupling  bends  rays  emanating  from  each  pixel 
of  each  subdisplay  so  they  have,  from  the  viewer’s 
perspective,  a  continuous  pixel  pitch.  Micro-lenses  and 
micro-gratings  are  fabricated  by  standard 
photolithographic  techniques.  One  implementation 
might  be  a  sheet  of  miniature  optical  elements,  one  per 
pixel.  Cox  at  Honeywell  in  Bloomington  MN  and 


Shvartsman  at  DuPont  in  NJ  have  teamed  to  develop 
such  a  microlens  array  which  might  be  applied  to  tile 
any  flat  panel.2*  A  hologram  might  also  be  used  to 
implement  the  same  stitching  effect  as  provided  a  sheet 
of  optical  elements. 

Marrakchi  et  al.29  at  Bellcore  in  Red  Bank,  NJ  are 
using  two-dimensional  SLMs  to  make  micro-lense  and 
micro-grating  arrays  programmable.  The  reconfigurable 
optical  crossconnection  technology  may  help  in  the 
optical  stitching  method  of  tiling  flat  panels. 

3.2.4  Combined  technologies 
Buzak30  of  Tektronix  in  Beaverton,  OR  is  developing 
TekVision™,  a  plasma  addressed  liquid  crystal  (PALC) 
display  technology.  In  this  novel  design,  the  active 
matrix  addressing  is  implemented  with  ionized  gas 
(plasma)  confined  in  channels  rather  than  the  more 
commonly  used  TFT,  MIM,  or  diode.  Plasma  was  used 
to  address  polymer  dispersed  liquid  crystal  material  in 
early  work,  but  twisted  neumatic  liquid  crystal  material 
is  used  now  for  the  direct  view  application.  A  PALC 
panel  comprises  the  LC  and  a  protective  dielectric  layer 
between  upper  and  lower  glass  sheets.  The  upper  sheet 
is  patterned  with  data  electrodes;  the  lower,  with 
channels  (grooves)  containing  two  parallel  electrodes  for 
anode  and  cathode.  The  objective  is  the  definition  and 
development  of  high  volume  manufacturing  techniques 
for  displays  characterized  by  fiat  panel  depth  <  5  cm  (2 
in.),  large  image  size  >  38  cm  (15  in.)  diagonal,  24  bit 
full  color,  and  >16  kHz  line  rates  to  support  video. 
Work  is  currently  underway  on  a  40  cm  (16  in.) 
diagonal  demonstration  panel.  Because  of  the  simple 
electrode  structure  with  all  lines  running  edge-to-edge 
across  the  complete  display,  PALCs  should  be  readily 
scalable  to  PCCADS  size  (2000  cm2). 

4.  PROJECTION 

There  are  several  projection  methods:  cathode  ray  tube 
(CRT),  liquid  crystal  light  valve  (LCLV),  deformable 
mirror  device  (DMD)  chips,  active  matrix  liquid  crystal 
display  (AMLCD),  lead  lanthanum  zirconium  titanate 
(PLZT)  devices,  electron-beam-addressed  membrane 
light  modulators  (e-MLM),  and  solid  state  laser  display 
(SSLD).  Rlaha31  has  recently  reviewed  other  large 
screen  display  technologies.  Projector  light  power 
output  is  given  in  watts  at  the  aperture  or  lm  at  the 
screen;  one  must  specify  a  projection  solid  angle  to 
compute  luminance  or  a  screen  size  to  compute 
illuminance. 

4.1  CRT  and  LCLV 

Cathode  ray  tubes  and  fight  valves  have  been  used  in 
projectors  for  some  time.  The  CRT  displays  are  of 
much  lower  quality  than  the  LCLV.  Militarized 
versions  of  both  technologies  are  expensive,  heavy. 
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power-hogs  of  low  reliability.  Present  applications 
include  command/control  centers  and  simulators.  An 
example  is  the  new  Hughes  Series  300  LCLV  system, 
based  on  an  optically  written  a-Si  photoconductor, 
which  projects  2S00  lm  at  video  rate  with  good  contrast 
for  a  price  of  $150,000;  additional  limitations  include 
low  frame  rates  and  thermal  sensitivity. 

4.2  DMD 

Deformable  mirror  devices  present  a  near  term  practical 
alternative  to  the  CRT  and  LCLV  projectors  for 
applications  not  requiring  small  weight  and  space. 
Younse  and  coworkers32  at  Texas  Instruments  (TT)  in 
Dallas,  TX  is  developing  DMD  technology  for  a  variety 
of  products,  including  printers  and  high  definition 
displays.  A  768  x  576  pixel  PAL  DMD  chip  has  been 
fabricated,  and  a  PAL/NTSC  live- video  projector  will  be 
demonstrated  en  route  to  the  high  definition  display. 
Early  test  chips  show  excellent  grayscale  and  color  and 
a  140:1  contrast  ratio;  the  breadboard  optics  system  is 
delivering  560  lm  monochrome  B/W  to  the  152  cm  (60 
in.)  diagonal  screen  of  area  about  1  m2,  or  about  560  lx. 
Color  is  presently  achieved  using  frame  sequential 
operation  of  the  DMD  and  a  rotating  color  filter  wheel, 
bringing  color  illumination  to  about  185  lx.  A  3-DMD 
color  optics  design  is  being  considered.  The  prototype 
projection  optical  subsystem  with  a  500  W  source 
producing  >1000  lm  is  being  developed  by  DSRC.  The 
mechanical  durability  of  the  minor  hinges  is  being 
investigated  by  Pryputniewiez33  at  Worchester 
Polytechnic  Institute  in  Worchester  MA.  The  prototype 
1940  x  1080  pixel,  high  definition  display  system 
incorporating  a  3.2  x  1.9  cm  (1.25  x  0.75  in.)  DMD 
chip  with  a  16:9  aspect  ratio,  a  150:1  contrast  ratio,  and 
projecting  >1000  lm  to  the  screen  is  to  be  demonstrated 
in  late  1993.  DMD  projectors  may  find  applications  as 
transport  and  bomber  aircraft  PCD  prior  to 
miniaturization  of  the  light  source. 

4 3  AMLCD 

Liquid  crystal  displays  can  be  used  in  projection  as  well 
as  direct  view.  The  overall  efficiency  requires 
improvement  and  is  being  addressed  using  a  prototype 
LCD  projector  by  Glenn34  at  Florida  Atlantic 
University  in  Boca  Raton,  FL;  a  high  resolution,  high 
brightness  color  projector  is  scheduled  to  be 
demonstrated  in  1994;  lenticular  lens  plates  are 
registered  with  pixels  to  improve  efficiency  (by  a  factor 
of  4)  and  to  reduce  heating.  Xerox  has  announced  a 
525  x  525  pixel,  3.8  cm  (1.5  in.)  square,  p-Si  on  quartz 
AMLCD  for  projector  applications.  DSRC  is 
considering  the  application  its  seif-scanned  high  density 
(800  dpi)  AMLCD  technology,  which  involves  p-Si  on 
glass  and  quartz  substrates,  to  projectors.33 
Schwarzer34  at  Hughes  in  Carlsbad,  CA  has  developed 
the  Hughes  HighBright™  cockpit  display  based  on  three 


a-Si  AMLCDs  operating  in  a  color  projector  design;  the 
breadboard  system  is  sunlight  readable  and  has  an  active 
display  area  of  16  x  16  cm  (6.25  x  6.25  in.).  The  size, 
weight,  and  power  requirements  of  the  light  sources  for 
these  projectors  all  need  reduction;  they  are  brighter 
than  CRTs,  but  otherwise  may  provide  little 
improvement  at  present 

4.4  PLZT  and  e-MLM 

The  LCLV,  DMD,  and  AMLCD  are  all  spatial  light 
modulators  (SLMs).  Additional  SLM  designs  are  being 
considered  for  the  projector  application.  Gobeli37  at 
Foresight,  Inc.  in  Winter  Park  FL  is  evaluating  PLZT  as 
the  basis  of  a  projection  display  and  may  develop  a  high 
brightness,  high  resolution  display  based  on  PLZT. 
Ward  and  coworkers3*  at  Option  Systems,  Inc.  in 
Bedford,  MA  have  studied  the  application  of  their  e- 
MLM  device  to  projectors  and  conclude  that  a  1280  x 
1024  pixel  monochrome  prototype  would  deliver  2000 
lm  to  a  >132  cm  (52  in.)  diagonal  screen  with  a  contrast 
ratio  of  300:1  with  a  60  Hz  frame  rate.  Considerable 
work  remains  before  such  technologies  might  be 
considered  for  cockpit  applications.  For  example,  the  e- 
MLM  approach  would  require  three  of  everything  to 
achieve  color,  including  high-brightness  lamps,  c-MLM 
devices,  and  optics. 

4.5  Laser 

Lasers  present  a  very  promising  technology  for 
achieving  panoramic  cockpit  displays.  Laser  light  is 
coherent  and  colors  are  fully  saturated.  The  coherency 
translates  to  a  unique  feature  of  laser  displays:  virtually 
infinite  depth  of  focus.  This  means  that  the  image  is 
always  in  focus,  even  when  displayed  on  curved  or 
domed  screens,  as  in  a  custom  installation  inside  a 
cockpit  or  simulator.  The  pure  colors  provide  a  wide 
color  spectrum  capability.  The  color  range  is  larger 
than  CRT  or  LCD  based  systems.  Furthermore,  a  laser 
display  has  better  legibility:  objects  which  are  fuzzy  in 
a  CRT  or  LCD  system  are  clear  in  a  laser  projection  of 
the  same  image  size.  Laser  display  technological 
approaches  include  discrete  lasers  (both  gas  and  solid 
state),  laser  arrays  (solid  state),  and  a  CRT  having 
semiconductor  materials  in  place  of  phosphors.  The 
various  solid  state  approaches  vary  in  the  pumping 
mechanism. 

4.5.1  Gas/Dye 

A  gas/dye  laser  projector  was  developed  by  Visulux  in 
1986  for  the  U.S.  Air  Force.33  The  system  was 
designed  to  produce  an  image  size  from  6-12  m  (20-40 
feet)  wide  and  have  a  lifetime  of  10000  hr  at  3000  lm 
equivalent  incoherent  light  brightness.  Some  18  of 
these  large  screen  display  systems  were  sold  at  about 
$250K  each;  most  were  uccd  in  command/control 
centers.  HQ  MAC  used  the  system  to  produce  a  3.67  x 
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9.S  m  (12  x  32  ft)  picture  of  the  world  and  to  project 
where  all  its  airplanes  were;  when  they  doubled  the  size 
of  an  area  on  the  map  they  got  the  same  resolution  and 
picture  quality  all  across  the  screen.  These  projectors 
are  remarkable  in  their  performance  but  require  too 
much  space,  power,  weight,  and  cooling  for  use  as  an 
airborne  display.  The  projector  weighs  270  kg  (600  lb) 
in  a  space  of  1.4  m1  (48  caft.),  and  runs  on  3  phase  480 
VAC  power.  In  addition  to  command  and  control 
centers,  such  projector  systems  may  also  be  appropriate 
for  use  in  simulators  and  other  ground-based  platforms 
where  the  support  requirements  of  gas/dye  lasers  are  not 
prohibitive. 

4.5.2  Solid  State 

Solid  state  lasers  (SSLs)  present  an  attractive  method  of 
achieving  a  PCD.  The  three  primary  colors  are  to  be 
produced  by  direct  SSL  lasing  at  the  visible  frequency 
where  possible,  or  else  by  frequency  doubling  of  near 
infrared  SSLs.  Such  a  device  might  require  but  0.025 
m3  (<1  cu.ft)  and  be  some  5%  efficient  in  converting 
electrical  power  to  light  on  the  screen. 

Nitor  Corporation  in  San  Jose,  CA  is  pursuing  an 
approach  based  on  red,  green,  and  blue  semiconductor 
lasers  with  optimized  source  wavelengths,  solid  state 
modulation,  and  electromechanical  scanning.40  A  red 
laser  at  640  nm  has  been  developed,  blue  has  been 
demonstrated,  and  green  is  under  development.  Each  of 
the  primary  colors  (red,  green,  and  blue)  is  modulated 
and  combined  into  white  (tri-color)  light  and  raster- 
scanned  across  the  screen.  The  optical  design  has  been 
simulated,  and  the  electronics  design  is  complete.  The 
schedule  is  for  monochrome  NTSC  and  RGB  high 
definition  prototypes  by  1993.  The  color  prototype  is  to 
project  up  to  1920  x  1035  tri-color  pixels  in  an  image 
size  of  1-5  m  (50-200  in.)  diagonal,  have  a  contrast 
ratio  of  150:1,  require  wall  plug  power,  have  a  lifetime 
of  40000  hr,  and  have  a  brightness  (equivalent 
incoherent  light)  of  1800  lm.  The  Nitor  solid  state  laser 
projector  design  is  summarized  in  Figure  14. 

Several  SSL  approaches  are  being  pursued  for 
production  of  all  three  primary  colors.  One  class 
involves  rare  earth  doped  garnets  pumped  with  infrared 
diodes.  McFarlane41  at  Hughes  in  Malibu,  CA  is 
performing  materials  research  necessary  for  this  method 
to  proceed.  Hughes  is  working  on  visible  upconversion 
lasers  and  plans  to  make  an  all  solid  state,  diode 
pumped  RGB  laser  system  for  projection  and  other 
applications;  0.5  W  has  been  achieved  at  551  nm  with 
pumping  at  797  nm,  and  1.1  W  is  projected.  Another 
class  of  visible  laser  technology  is  the  vertical  cavity 
surface  emitting  laser  (VCSEL)  based  on  molecular 
quantum  wells  structures  fabricated  from  GaAlAs/GaAs. 
Also,  laser  diode  arrays  are  already  used  to  produce 


sufficient  power,  albeit  in  the  infrared.  These  latter 
approaches  can  produce  red  directly  but  requires 
frequency  doubling,  or  other  inefficient  energy 
conversions,  to  achieve  green  or  blue. 

Significant  research  remains  before  visible  solid  state 
lasers  are  ready  for  displays.  Blue  is  the  biggest 
challenge:  only  10-100  mW  of  light  output  has  been 
demonstrated  to  date.  Harrison42  at  Schwartz  Electro- 
Optics  in  Concord,  MA,  for  example,  has  studied  the 
second  harmonic  generation  problem  for  DARPA  using 
a  frequency  doubled,  diode  laser  pumped  Nd:YAG  laser 
operating  at  473  nm  and  concluded  that  alternative 
configurations  must  be  considered  to  achieve  sufficient 
power  at  a  blue  wavelength. 

4.5.3  E-Beam  Pumped  Semiconductor 
Another  full  color  SSLD  method  is  an  e-beam  pumped 
semiconductor  (EBS)  laser  fabricated  as  a  two- 
dimensional  array  in  a  CRT  with  II- VI  materials  (CdSe- 
red,  CdS-green,  ZnSe-blue)  in  place  of  the  phosphors. 
Such  devices  provide  laser  emission  from  more  than 
1000  x  1000  randomly  addressable  spots.43  Other 
materials,  including  GaAs,  are  being  tested  owning  to 
the  poor  lifetime  characteristics  of  the  II- VI  materials. 
Bhargava  et  al.44  at  North  American  Philips  in 
Briarcliff  Manor,  NY  have  reviewed  this  approach  for 
large  area  projection  TV  displays.  Rice43  at 
McDonnell  Douglas  in  St  Louis,  MO  is  considering 
developing  the  RGB  laser  video  projector  developed 
within  the  last  two  years  by  Nasibov  et  al.  at  the 
Lebedev  Institute  in  Moscow,  Russia.  The  EBS 
approach  might  be  made  useable  in  a  cockpit  if  the 
pumping  were  provided  by  a  matrix  addressed  variation 
of  the  flat,  micro-tip  field  emission  plate  currently  being 
considered  for  application  in  LCD  backlights. 

5.  SMALL  FORMAT  DISPLAYS 
Research  is  presently  underway  to  establish  a  small 
format  high  resolution  display,  or  smafhired,  technology 
capable  of  presenting  video  rate  imagery  (real  or  virtual) 
in  applications  requiring  ultra  low  weight  and  power, 
such  as  a  helmet.  A  1280  x  1024  pixel  monochromatic 
full  grayscale  smafhired  would  satisfy  some  80%  of  the 
identified  DoD  requirment  for  HMDs  if  the  active  area 
were  2.5  cm  (1  in.)  square  in  a  5  cm  (2  in.)  package.46 
For  future  applications  it  is  noted  that  the  IFOV 
required  to  support  ILO  systems  is  120  x  60°  with  a  4pi 
sr  FOR  with  a  resolution  of  1  mrad.  Applications  for 
smafhireds  include  not  only  military  HMDs,  but  also 
commercial  (personal  video,  telecommunications,  arcade 
games,  computer  displays,  etc.). 

5.1  PCCADS  HMDS 

Current  HMDS  technology  permits  the  pilot  to  use  head 
pointing  to  position  sensors  and  seekers  in  directions 
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outside  of  the  HUD  FOV,  thereby  giving  him  quicker 
first  shot  capability.  The  Agile  Eye  system  produced  by 
Kaiser  was  used  by  McDonnell  Douglas  in  the 
PCCADS  program.  The  lead  pilot  had  a  12°  IFOV  to 
designate  visual  targets  and  to  provide  HUD  type 
information  on  the  helmet  visor.  The  pilot  FOR  was 
that  of  his  head  range  of  modern.  A  1.3  cm  (O.S  in.) 
diameter  CRT,  the  high  voltage  supply  for  the  CRT,  the 
optical  elements  necessary  to  project  the  CRT  image 
onto  the  visor,  and  the  head  tracker  receiver  were 
incorporated  into  the  helmet  An  HMDS  system  is 
under  consideration  as  a  growth  capability  for  the  F-22 
air  superiority  fighter.  A  day  only  HMS  is  being 
developed  for  the  F-16  as  a  retrofit  item.  The  Sextant 
Avionique/Intertechnique  lightweight  helmet  with 
integrated  sight  and  display  has  been  selected  for  use  on 
the  French  Rafale  combat  aircraft47  These  systems 
represent  today’s  technology  and  provide  graphics  but 
no  video. 

Smafhireds  will  provide  video.  Because  the  smartiired 
is  mounted  inside  an  HMD,  43000-65000  lx  (4000-6000 
fL)  luminance  is  required  for  the  most  demanding 
application.4*  Less  is  required  at  night  or  at  such  times 
as  the  canopy  is  opaqued,  but  most  mission  segments 
will  be  flown  with  clear  canopy  to  take  advantage  of 
that  fantastic  real  world  display  provided  by  nature. 
Accordingly,  micro  CRTs,  AMELDs,  LEDs,  etc.  may 
compete  successfully  with  AMLCDs  and  SSLDs  in  this 
application  only  if  they  too  can  satisfy  the  sunlight 
readibility  criterion. 

52  Miniaturized  CRT 

Smafhireds  presently  take  the  form  of  the  miniaturized 
CRTs  developed  over  the  past  15  years  under  the 
Armstrong  Laboratory  computer  created  view  research 
program.  These  devices  are  still  too  heavy  for  a  fighter 
pilots  pulling  high  G-loads  and  soldiers  fighting  for 
hours.  System  integrators  such  as  Kaiser49  of  San 
Jose,  CA  and  S-TRON30  of  Mountain  View,  CA  are 
evaluating  small  format  CRTs,  but  are  looking  to  flat 
panel  and  other  technologies  for  less  weight  and  power 
with  more  resolution  and  color. 

53  Flat  Panel 

Smafhireds  might  be  realized  with  much  the  same  flat 
panel  technology  as  for  HDDs.  A  program  to 
miniaturize  flat  panels  for  application  in  military  avionic 
HMD  systems  is  being  undertaken  by  this  office;  our 
projected  emphasis  is  the  AMLCD  approach.  The 
Army  is  working  on  smafhireds  under  its  Soldier’s 
Integrated  Protective  Ensemble  (SIPE),  Combat  Vehicle 
Crew  (CVQ,  and  other  programs.31  System 
integrators  are  considering  the  use  of  lighter  weight  flat 
panel  displays  instead  of  CRTs  as  smafhireds  in  HMDs. 
Kopin  is  developing  x-Si  AMLCD/ AMELD  smafhired 


technology.  Also,  DSRC  is  considering  development  of 
the  necessary  LCD  technology;  Planar  of  Beaverton, 
OR,  the  ELD  technology. 

5.4  Laser  Retinal  Painter 

The  terminology  "laser  retinal  painter"  is  hereby 
introduced  to  cover  a  new  class  of  HMD:  an  image 
generated  by  a  very  low  power  laser  beamed  through 
the  pupil  and  scanned  across  the  retina.  The  scanning 
and  modulation  of  the  laser  is  controlled  by  solid  state 
devices,  such  as  acousto-optic  modulators.  Furness32 
at  the  University  of  Washington  in  Seattle  began 
experiments  on  such  a  system  in  January  1992. 

5.5  Visor  Projection  and  Other  Approaches 

A  variety  of  other  approaches  are  being  explored  to  take 
smafhireds  from  concept  today  to  technology  tomorrow. 
Many  of  these  approaches  involve  projection  on  the 
inside  surface  of  the  helmet  visor.  The  projection 
concept  is  as  follows.  The  system  tracks  the  pupil  as 
well  as  the  head  position  and  projects  high  resolution 
information  in  the  center  5°  HIFQV  for  direct  vision  and 
low  resolution  information  into  the  full  120“  IFOV 
peripheral  vision.  The  content  of  the  IFOV  is  based 
primarily  on  head  position;  the  content  of  the  HIFOV, 
pupil.  Motion  cues  in  the  IFOV  are  used  to  direct  the 
viewers  HIFOV  scan  of  the  virtual  reality  presented  by 
the  smafhired  system.  The  HDD  projector  approaches 
discussed  in  Section  5  above  might  all  be  miniaturized, 
in  principle,  to  provide  a  variety  of  smafhired  projectors 
for  evaluation  in  PCD  systems. 

6.  MULTIPERSPECTIVE  CAPABILITY 
Part  of  our  vision  for  2000  is  a  capability  of  the  display 
to  operate  in  a  multiperspective  mode.  That  is,  the 
display  can  be  switched  to  show  terrain  or  threat  data, 
for  example,  in  full  3-D.  This  capability  will  increase 
pilot  situational  awareness  by  providing  realistic  3-D 
display  of  information  to  the  pilot  when  needed. 
Multiperspective  means  hologram,  or  hologram-like, 
enabling  the  viewer  to  look  around  objects  and  to  view 
them  from  a  wide  range  of  angles.  Aviation 
applications  have  been  reviewed  by  Wickens,  Todd,  and 
Seidler.33  Hopper54  has  reviewed  hardware 
approaches  to  3-D  which  have  promise  of  being 
eventually  applied  in  a  cockpit  Some  examples  follow. 

6.1  Autostereoscopic  Flat  Panel 
Eichenlaub55  at  Dimension  Technologies,  Inc.  (DTI)  in 
Rochester,  NY  is  advancing  an  LCD  based 
autostereoscopic  display  technology  for  the  Air  Force. 
Two  perspectives,  one  per  eye,  are  produced  using 
stereoscopic  display  format  generators  and  a  head 
tracker  is  used  to  determine  the  instantaneous 
perspective  set  sent  to  the  HDD.  The  DTI  display 
device  is  capable  of  displaying  real  time  video 
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information,  as  well  as  computer  input,  and  is 
switchable  between  3-D  and  2-D  modes.  The  first 
product  has  been  developed:  a  stand-alone  640  x  480 
color  pixel  3-D  LCD  which  accepts  input  from  IBM  PC, 
Apple  MAC,  or  TV  cameras.  It  will  produce  8  bit, 
32768  ctdos  from  200W  or  less  at  30  frames  per 
second  on  a  30  cm  (10.2  in.)  diagonal  screen  which  is 
easily  readable  in  108000  lx  (10000  fc)  ambient  light  at 
a  contrast  ratio  of  100:1.  The  head  tracking  is  soon  to 
be  upgraded  from  electromechanical  to  electro-optical. 
A  second  product  in  development  will  provide 
autostereo  capability  and  full  horizontal  resolution  to 
both  eyes.  Work  on  a  high  resolution  display  has  begun 
to  enable  the  viewer  to  see  the  full  resolution  from  all 
locations,  thereby  providing  multiperspective  capability 
for  PC  D-sized  displays.  The  DTI  autosteroscopic  3D 
display  is  illustrated  in  Figure  IS. 

62  Volumetric 

Laser  addressed  volumetric  displays  provide  another 
route  to  multiperspective  cockpit  displays.  The  enabling 
technology  is  being  developed  by  Williams  and 
Garcia54  at  Texas  Instruments  in  Dallas,  TX 
(OmniView™)  and  by  Sol  tan37  at  the  Naval  Ocean 
Systems  Center  in  San  Diego  under  joint  Navy  and  Air 
Force  funding.  In  both  cases  lasers  project  the  image 
onto  a  rotating  helix  screen  and  points  are  plotted  in  X, 
Y,  Z  space  to  provide  a  real  spatial  image.  All  possible 
perspectives  (3-D  views)  are  provided  in  a  single  spatial 
image  (one  just  moves  around  the  display  object).  The 
largest  TI  custom  volumetric  display  addresses  4096  x 
4096  x  4000  volume  elements  (voxels)  of  2.5  mm  size 
contained  in  a  cylinder  0.91  m  in  diameter  by  0.46  m 
height  (36  in.  D,  18  in.  H).  Some  4000  voxels  per 
color  can  be  projected  at  any  one  time:  4000  white 
display  voxels;  or  4000  each  of  red,  green,  and  blue  for 
a  total  of  12000  color  display  voxels.  The  NOSC 
display  under  development  has  as  objectives  the 
projection  35000  voxels  in  a  33  cm  (13  in.)  diameter 
double  helix  use  solid  state  lasers  on  a  3-D  display 
media  with  no  moving  parts.  A  charactericstic  of  these 
volumetric  displays  is  that  there  are  no  hidden  lines:  all 
image  formats  must  be  designed  to  be  viewable  from  all 
directions  with  no  hidden  line  removal. 

63  Hologram  Projection 

Benton  and  coworkers5*  at  the  the  MIT  Media 
Laboratory  in  Cambridge,  MA  are  researching  the 
display  of  computer  generated  holograms  (CGHs)  of  3- 
D  objects.  Hologram  sequences  have  been  computed  on 
the  Connection  Machine  Model  2  supercomputer  and 
transmitted  via  a  data  network  to  the  laboratory 
computer  for  playback  with  a  laser  through  a  spatial 
light  modulator  (SLM)  having  one  physical  dimension. 
Vertical  parallax  is  sacrificed  to  speed  up  the 
computations  and  make  it  possible  to  project  the 


holograms  line  by  line  through  acousto-optic  modulator 
(AOM)  cells.  Good  quality  images  which  exhibit 
excellent  color  registration  can  be  projected  by 
illuminating  3  AOMs  with  633  nm  (red),  514  nm 
(green),  and  442  nm  (blue)  laser  light  A  6  megasample 
holographic  pattern  (one  frame)  can  be  computed  in  <ls 
from  an  image  comprising  1000  points  with  arbitrary 
intensity.  Farhoosh  et  aL59  have  used  an  SLM  having 
two  physical  dimensions  comprising  an  LCLV  in 
contact  with  a  CRT  via  a  fiber-optic  plate  to  achieve  a 
3  frame  holographic  movie  with  a  display  rate  of  1 
frame  per  second  Many  years  of  work  remain  before 
hologram  projection  technology  might  be  available  for 
cockpit  application. 

7.  SYSTEM  CONSIDERATIONS 
The  PCD  is  part  of  the  cockpit  displays  and  control 
system.  A  small  supercomputer,  known  as  a  graphics 
processor  (GP)  is  necessary  to  drive  such  a  large  area 
electronic  display  system;  similarly,  special  processors 
are  required  to  implement  pilot  decision  aiding  systems. 
Clearly,  the  functions  and  screen  formats  must  be 
determined  for  this  new  class  of  displays. 

7.1  Graphics  Processor 

A  supercomputer  in  a  shoebox  is  required  to  drive  die 
PCD  displays.  All  information  must  be  integrated  in 
standard  formats  and  graphic  generated  for  the  large 
area  of  high  resolution  display  surface(s).  This  office  is 
developing  this  display  engine  under  a  program  entitled 
Cockpit  Display  Generator  (CDG),  comprising  several 
contractual  and  in-house  research  projects.  Myers40 
has  published  an  overview  of  the  program.  The 
definition  phase  has  just  been  completed.41  In  addition 
to  electronic  processing  circuits,  optical  processing 
circuits  may  be  employed  to  reduce  the  size  and  weight 
of  the  on-board  processor.  Hopper42  has  reviewed 
international  work  in  optical  computing  and  processing. 

12  Information  Processor 

Pilot’s  Associate  (PA)  is  the  information  processing  aid 
which  determines  which  data  is  sent  to  the  graphics 
processor  and,  thence,  into  the  aforedescribed  display 
formats.  Banks  and  Lizza43  have  explored  the  lessons 
learned  in  building  a  cooperative,  knowledge-based 
system  to  help  pilots  make  decisions. 

13  Display  Format 

Reising  and  coworkers64  have  investigated  the 
application  of  color  graphics  and  large  screen  flat  panels 
to  information  display  in  military  cockpits.  They 
strongly  advocate  use  of  color  pictorial  formats  to 
provide  intuitive  presentation  and,  thereby,  a  potential 
reduction  in  pilot  workload.  A  large  display  is  critical 
to  integrate  all  information  in  a  meaningful,  legible  way. 
Future  electronic  multifunctional  displays  (MFD)  must 
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deliver  both  color  and  large  area  to  support  the  display 
format  requirements.  Reising  and  Emerson  note  that 
one  day  tire  entire  instrument  panel  may  consist  of  one 
display  surface  where  both  pictorial  and  alpha-numeric 
formats  will  be  displayed.  The  large  HDD  system  may 
be  augmented  with  a  virtual  cockpit  where  the 
instrument  panel,  as  it  is  known  today,  is  replaced  by 
images  of  the  instruments  projected  on  the  pilot's  visor. 
They  also  recommend  three-dimensional  (3D)  capability 
for  some  display  formats. 

8.  DISCUSSION 

Modelling  efforts  will  be  necessary  to  evaluate  the  new 
PCD  technologies  as  they  evolve  against  requirements, 
and  fields  other  than  aircraft  must  be  monitored  for 
synergistic  developments. 

8.1  Hierarchical  Modelling/Simulation 

A  set  of  hierarchical  models  are  being  developed  for  use 
in  display  design  and  evaluation.  The  technologies 
described  here  can  be  used  in  a  myriad  of  ways  against 
PCD  objectives.  Modelling  includes  simulation  and 
will  reduce  the  number  of  costly  fabrication  and  testing 
projects  necessary  to  evaluate  the  possibilities  against  all 
key  user  considerations  (functionality,  compatibility, 
producibility,  availability,  reliability,  maintainability, 
affordability).  This  is  a  complexity  analysis  problem  and 
requires  hyperspace  modelling.  The  display  level 
(functionality)  model  comprises  submodels  of  both  man 
and  machine;  the  development  of  such  a  model  is  being 
performed  by  a  multi-institutional  team  lead  by  Larimer 
at  NASA  Ames  at  Moffett  Field,  CA.65  Demand 
models  will  track  the  ever-changing  requirements  for 
sunlight-readable  displays  in  military  systems; 
development  of  this  model  is  being  lead  by  Hopper.66 
The  producibility  model  contains  representations  for 
processes,  equipment,  operator  training  levels,  etc.;  one 
widely  available  model  is  that  produced  by  Resor67  at 
MRS  Technology  Inc.  in  Chelmsford,  MA. 
Breadboards  at  several  hundred  thousand  dollars  each, 
nor  factories  at  several  tens  to  hundreds  of  millions 
each,  can  be  built  just  to  try  out  the  possibilities. 
Modelling  is  much  more  effective  apd  will  become  our 
display  evaluation  tool  for  the  1990's  and  beyond  to 
maximize  what  we  learn  from  each  breadboard  and 
flight  test  and  to  minimize  the  number  which  are 
necessary.  Accordingly,  a  hierarchical  modelling  effort 
incorporating  all  appropriate  models/simulations  is  being 
undertaken  for  cockpit  applications  by  our  office. 

8.2  Synergistic  Applications 

Other  applications  exist  in  the  area  of  optical  processing 
and  computing  for  the  smafhireds  needed  for  HMDs  and 
for  the  spatial  light  modulators  used  in  some  projector 
and  3-D  display  approaches  Advancements  in  such 
areas  will  expedite  the  development  of  PCD  technology. 


9.  CONCLUSIONS 

The  advantages  of  a  large  area  display  system  were 
recently  demonstrated  in  the  Panoramic  Cockpit  Control 
and  Display  System  program,  a  joint  research  effort  of 
the  Wright  Laboratory  and  Armstrong  Laboratory.  The 
key  objective  result  was  a  43%  increase  in  pilot  combat 
effectiveness,  which  translates  to  a  31%  reduction  in  the 
number  of  aircraft  and  pilots  needed  for  a  given 
mission.  The  price  is  the  development  of  display 
hardware  necessary  to  put  such  display  systems  into 
operational  aircraft.  As  airplanes  and  pilots  cost  several 
orders  of  magnitude  more  than  the  research  and 
development  program  required,  significant  cost  savings 
will  result-more  defense  for  less  money. 

An  array  of  AMLCDs  has  provided  an  approximation  of 
the  panoramic  display  concept  Such  an  array  is  the 
realization  of  our  present  cockpit  vision.  Several 
systems  now  in  EMD  have  selected  this  approach, 
including  the  F-22  air  superiority  fighter,  the  RAH-66 
light  attack  helicopter,  and  the  C-130/C-141  transport 
retrofit  program. 

The  1996  vision  will  most  likely  be  realized  by  an 
evolution  of  present  AMLCD  technology  to  larger,  625 
cm2,  areas  per  display  combined  with  tiling  techniques. 
Such  techniques  are  a  natural  extention  of  todays 
technology  and  are  expected  to  dominate  fra  the 
foreseeable  future.  Autos tercoscopic  3-D  technology 
will  be  available  for  use  with  flat  panel  displays. 

Solid  state  lasers  present  what  is,  perhaps,  the  most 
attractive  alternative  for  achieving  panoramic  cockpit 
display  technology  by  2000.  They  are  light  in  weight, 
low  in  power  requirements,  and  can  meet  all 
environmental  requirements  more  readily  than  the  other 
methods  discussed  Further  more,  they  can  scale  from 
HDD  down  to  HMD  sizes  and  up  to  dome  size.  Also, 
applications  of  projectors  extend  beyond  to  aircraft 
cockpits  to  simulators  and  training  aids.  Small  format 
display  technology  requires  significant  research  before 
high  resolution  images  in  helmet  systems  become  a 
design  option.  Smafhireds  with  1280  x  1024  pixels  will 
be  available  for  application  as  a  monochrome  HMD 
with  60°  x  60°  IFOV  and  as  color  projector  for  HDD 
instruments.  Multiperspective  (true,  full,  look-around  3- 
D)  technologies  will  be  available  for  application  in 
HDD  systems;  the  HMD  system  will  be  capable  of 
providing  autostereoscopic  (one  perspective  per  eye) 
formats.  Multiperspective  display  development  is  fast 
making  true  3-D  a  cockpit  design  option. 

By  2020  a  variety  of  projector  and  direct  view 
technologies  will  be  available  to  build  HDD  and 
encapsulated  cockpit  display  systems.  Individual  HDDs 
will  be  16  million  (e.g.  4096  x  4096)  color  pixels  in 


2000  cm2  (300  in2)  and  contoured  to  fit  the  curved  x  2048  monochrome  pixels  (1.02  mrad  spots),  and  full 

surfaces  of  the  control  panel  and  inner  canopy.  Several  color  capable.  Both  the  HDD  and  HMD  system  will 

of  these  HDDs  will  be  tiled  to  achieve  larger  display  provide  multiperspective  3-D  capability  where  useful  in 

areas.  Smafhireds  will  be  120°  x  120°  IFOV  with  2048  the  cockpit  display  format  design. 
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Figure  3.  PCCADS  2000  displays. 
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Figure  4.  PCCADS  test  cockpit:  improved  man-machine 
interface  via  large  display  plus  HMDS. 


9-20 


VISION 

COCKPITS  INTO  THE  21st  CENTURY 


Figure  5.  Joint  Cockpit  Office  visions  for  today,  1996,  and  2005. 

Note:  the  technological  capability  to  implement  the  2005  vision  is  now  expected  by  2000. 


Figure  6.  Panoramic  cockpit  concept  for  beyond  2000 
with  retention  of  a  bubble  canopy. 


Figure  7.  Encapsulated  cockpit  concept 
(courtesy  of  McDonnell  Douglas). 
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Figure  10.  Schematic  drawing  of  RAH-66  displays  for  EMD. 
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Figure  1 1 .  Photograph  of  C-130  RAMTIP  cockpit  displays. 


Figure  12.  Litton  CdSe  AMLCD  used  at  navigator’s  station 
in  RAMTIP  with  drivers  and  connectors  shown. 


FULL  DISPLAY 
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100  sq  in  Display  Integral 

1200x800  Pixels  Row  Drivers 
Field  Sequential  «  320,000  Pixels 

3  Segments  ^|^M|^tOO  Pixels  in 


Figure  13.  Butt-coupling  of  p-Si  AMLCDs  to  create 
a  large  area  display  (CHDD  program). 
Note:  the  lead  count  has  been  further 
reduced  from  210  to  132. 


•  Integral  Drivers  Reduce  External 
Lead  Count  from  7200  to  210 
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Figure  14.  Solid  state  laser  projector  concept. 
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Discussion 


QUESTION  D.V.  GAGGIN 

Virtual  Cockpits  and  Panoramic  Displays  are  complementary  techniques. 
Does  the  U  S  Air  Force  have  a  program  that  will  bring  these  together  in  the 
near  future? 

REPLY 

Yes.  Panoramic  Displays  will  be  available  to  cockpit  designers  by  the 
year  2000.  Virtual  Cockpits  will  be  available  by  2020.  Both  large-area 
aircraft-mounted  and  helmet -mounted  display  systems  will  have  on  the  order 
of  3  million  pixels  for  each  eye.  The  graphics  processor  has  yet  to  be 
developed,  but  will  be  similar  for  each  of  these  types  of  display.  Thus 
the  development  of  panoramic  aircraft -mounted  displays  is  a  pre-requisite 
to  the  development  of  virtual  cockpits. 


QUESTION  0.  FOURURE 

The  concepts  of  Panoramic  Displays  and  Virtual  Cockpits  both  aim  at 
developing  synthetic  presentations  to  the  pilot,  with  geographic,  tactical 
and  system  status  information,  to  improve  situation  awareness.  Are  these 
two  concepts  compatible  with  each  other?  What  should  be  the  priority  in 
R&D  in  the  near  future? 

REPLY 

Concerning  priority,  panoramic  displays  for  the  year  2000  are  the 
higher  priority.  Technology  permits  that  realization  20  years  earlier  than 
the  virtual  cockpit. 

Concerning  compatibility,  the  two  concepts  are  compatible  in  the  sense 
that  the  development  of  the  panoramic  display  technology  will  come  first 
and  help  in  the  subsequent  development  of  the  virtual  cockpit.  Once  enough 
elements  of  the  Supercockpit  concept  (i.e.the  virtual  cockpit)  have  been 
developed,  it  will  be  possible  for  trade-off  studies  to  be  conducted 
between  aircraft -mounced  and  helmet -mounted  panoramic  displays.  This  will 
occur  in  the  period  2010  to  2020,  in  our  estimation.  Supercockpit  requires 
at  least  120  degees  *  60  degrees  high  resolution  video  color  from  the 
display  whether  mounted  on  the  aircraft  or  on  the  helmet.  A  100%  virtual 
cockpit,  as  described  by  Martin  (Paper  8)  would  obviate  the  need  for 
aircraft -mounted  displays,  provided  the  pilot  would  accept  it,  but  we 
believe  he  will  only  do  so  for  limited  mission  segments.  The  pilot  will 
want  to  see  through  the  helmet  display  and  fly  without  it  most  of  the  time. 


Additional  Note:  The  Supercockpit  requires  a  full-color  stereo 
binocular  helmet -mounted  display  having  2-4  million  pixels  per  eye. 
Such  displays  do  not  currently  exist.  We  are  just  beginning  development 
of  a  1  million  pixel  monochrome  display,  and  it  will  be  10  years  before  a 
display  of  the  full  capability  required  will  be  available  for  research. 
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VOL  AU-DESSUS  D  UN  MONDE  VIRTUEL 


P.  Larroque 
R.  Joannes 


DASSAULT  AVIATION,  Direction  G6n6rale  Technique 
BP  300  92214  SAINT-CLOUD  France 


1.  SOMMAIRE 

La  recherche  de  solutions  "discretes"  pour  les 
vols  de  penetration  A  trfes  basse  altitude  amene 
A  conslddrer  I'usage  des  fichiers  de  terrain,  de 
plus  en  plus  facilement  disponibles. 

Ces  considerations  ont  conduit  le  Ministere 
Franpais  de  la  Defense  A  confier  A  DASSAULT 
AVIATION  une  etude,  dans  le  cadre  du 
D6veloppement  Exploratoire  APIS  (Aide  au 
Pilotage  par  Imagerie  Synthetique),  relative  A 
■'utilisation  de  tels  fichiers  pour  la  constitution 
d'images  de  synth6se  A  bord  des  avions  de 
combat.  Ces  images  doivent  servir  A  aider  le 
pilote  A  conduire  sa  mission  par  lous  les  temps 
ou  la  nuit,  en  suppieant  la  vision  directe  du 
monde  exterieur. 

Cette  recherche  de  concept  a  utilise  en 
particulier  des  outils  de  simulation  pilot6e  temps 
r6el  qui  ont  permis  de  mettre  au  point 
differentes  figurations  APIS  en  t6te  haute  et  en 
tSte  moyenne. 

Cette  communication  pr6sente  la  demarche 
parcourue  dans  la  mise  au  point  de  ces 
figurations  avec  la  participation  des  Services 
Officiels  et  donne  des  exemples  des  propositions 
faites. 

Ces  images  ont  6t6  6valu6es  par  un  groupe  de 
dix  pilotes  comprenant  aussi  bien  des  pilotes 
d'essais  que  des  pilotes  des  Forces. 
L'6valuation  conduite  par  le  CERMA  a  mis  en 
6vidence  leur  interSt  ergonomique. 

Toutefois,  de  nombreux  points  demeurent  a 
r6gler  :  la  faisabilitg  technique  dans  des 
volumes  et  des  coQts  raisonnables,  les  aspects 
Ii6s  A  la  s6curit6  ou  A  la  fiabilit6  de  tels 
dispositifs... 


2.  SUMMARY 

Research  for  stealth  methods  of  low  level 
penetration  flight  leads  us  to  consider  the  use  of 
terrain  data  bases  which  are  becoming  more 
and  more  easily  available. 

This  is  why  the  French  Ministry  of  Defence  has 
granted  DASSAULT  AVIATION  a  contract  in  the 
scope  of  the  APIS*1*  Exploratory  Programme.  The 
purpose  of  this  study  was  to  consider  the  use  of 
terrain  files  for  designing  synthetic  images 
intended  for  combat  aircraft.  The  aim  is  to 
provide  the  pilot  with  pictures  replacing  direct 
sight  on  the  outside  world,  thus  helping  him  to 
conduct  the  flight  by  any  weather  or  at  night. 

The  research  for  this  new  concept  was 
supported  by  intensive  software  development  on 
real  time  simulation  tools.  The  latter  permitted 
the  proposal  of  different  APIS  representations  for 
both  head-up  and  head-level  displays. 

This  paper  presents  the  process  we  followed 
during  the  elaboration  of  the  images  with  the 
active  participation  of  the  French  Officials.  We 
also  give  some  examples  of  proposed  pictures. 

At  the  end  of  the  study  these  proposals  were 
assessed  by  a  team  of  ten  military  pilots, 
belonging  either  to  flight  test  teams  or  coming 
from  French  Navy  and  Air  Force.  This  evaluation 
has  been  carried  out  by  the  CERMA  (French 
Aero-Medical  Research  Institute). 

However,  many  points  are  still  to  be  examined  : 
how  to  display  these  pictures  onboard  while 
managing  safety  and  limiting  costs  and 
resources  consumption  at  an  affordable  level. 


W  This  French  abbreviation  means  :  Help  to  Pilot  using  Synthetic  Imageries 


3.  ABREVIATIONS 


APIS  Aide  au  Pilotage  par  Imagerie  Syn- 
thetique 

BPSA  Bureau  des  Programmes  de  Syst6- 
mes  d'Armes  (de  I'EMAA) 

CEAM  Centre  d'Experimentations  Aeriennes 

Militaires 

CERMA  Centre  d'Etudes  et  de  Recherches  de 

Mddecine  Aerospatiale 

CEV  Centre  d'Essais  en  Vol 

CTH  Collimateur  Tete  Haute 

CTM  Collimateur  T6te  Moyenne 

DE  D6veloppement  Exploratoire 

DGA  Delegation  Generale  6  I'Armement 

(du  Minist6re.de  la  Defense) 

DGAC  Direction  Generate  de  I'Aviation 

Civile 

EMAA  Etat-Major  de  I'Armde  de  I'Air 

HUD  Head-Up  Display  (terme  anglais  pour 

CTH) 

ILS  Instrument  Landing  System 

IMC  Instrument  Meteorological  Conditions 

(conditions  de  vol  aux  instruments) 

LAMAS  Laboratoire  de  M6decine  Aero¬ 

spatiale  (du  CEV) 

OASIS  Outil  d'Aide  6  la  Specification  des 

Informations  Syst6me 

OPE  Organisation  du  Poste  d'Equipage 

PERSEPOLIS  Programme  d'Etudes  et  de  Reali¬ 
sations  d'un  Systeme  Eiectronique 
Pour  I'Organisation  et  la  Lecture  d'ln- 
formations  Synthetiques 

STTE  Service  Technique  des  Telecom¬ 

munications  et  des  Equipements 
aeronautiques 

TBA  Tr6s  Basse  Altitude 

4.  INTRODUCTION 

Dans  la  conception  de  I'aviation  de  combat 
moderne,  les  appareils  doivent  posseder  de 

nombreuses  capacit6s  et  en  tout  premier  lieu 
celles  d'observer  ou  d'attaquer  les  positions  de 
I'ennemi  au  sol  quelquefois  assez  en  arriere  de 
la  ligne  des  combats.  Dans  toutes  les  forces 


aeriennes,  ces  missions  sont  differemment 
repertoriees  mais  on  trouve  en  general  les 
memes  grands  themes  ;  retenons  en  particulier 
I'attaque  d'objectifs  fixes  de  taille  et  de  valeur 
importantes  (usines,  etc),  la  destruction  de 
noeuds  de  communication  (ouvrages  d'art, 
tunnels  ..),  la  desorganisation  des  aerodromes 
(pistes,  radars,  eventueilement  aeronefs  au  sol) 
ou  encore  la  localisation  et  la  neutralisation  des 
defenses  sol-air. 

Pour  toutes  ces  missions,  un  grand  point 
commun  :  I'arrivee  au  voisinage  de  I'objectif  ou 
du  point  de  tir  se  fait  en  vol  de  penetration  6 
tr6s  basse  altitude  ("vol  TBA").  Seul  ce  type  de 
vol  permet  de  traverser  avec  succes  les 
defenses  sol-air  de  I'adversaire.  Selon  les 
differents  scenarios,  cette  phase  de  vol  varie  en 
durde  et  en  tolerance  sur  les  vitesses  et 
hauteurs  de  vol  6  pratiquer. 

La  tendance  generate  des  dernteres  annees  a 
ete  :  plus  vite,  plus  bas,  plus  longtemps.  Ainsi, 
la  ou  op6raient  avant  les  MIRAGE  lll-E,  doivent 
operer  maintenant  des  MIRAGE  2000-N,  dotes 
d'un  Systeme  de  Navigation  et  d'Armement 
offrant  des  modes  de  "suivi  de  terrain". 
Aujourd'hui,  ces  limites  de  vitesse  et  de  hauteur 
etant  sans  cesse  plus  difticiles  6  repousser,  on 
cherche  6  gagner  en  discretion.  C'est  ainsi  que 
le  "suivi  de  terrain"  6  base  de  radar,  classique 
pour  un  F  111  ou  un  TORNADO  (IDS),  doit 
evoluer.  C'est  ce  qu'il  a  d6j£  fait  dans  les 
MIRAGE  2000-N  et  les  tout  derniers  MIRAGE 
2000-D  ou  des  capacites  de  "vol  sur  fichier" 
existent  en  complement  aux  modes  utilisant  le 
radar  ANTILOPE. 

Le  "vol  sur  fichier"  exploite  toutes  les 
possibilites  des  techniques  modernes  en  mattere 
de  connaissance  du  relief.  La  constitution  de 
bases  de  donn6es  de  terrain  en  n'importe  quelle 
region  de  la  Terre  est  a  notre  port6e  grace  en 
particulier  aux  observations  par  satellite.  Les 
progr6s  en  mature  de  traitement  de 
('information  au  sol  et  aussi  en  vol  permettent 
de  stocker  6  bord  de  I'avion  les  zones  que  Ton 
se  propose  de  survoler. 

L'av6nement  de  ces  techniques  a  ainsi  conduit 
le  Ministere  Fran^ais  de  la  Defense  6  lancer  un 
Developpement  Exploratoire  sur  ce  theme. 
Denomme  APIS  (pour  Aide  au  Pilote  par 
Imagerie  Synthetique),  il  a  pour  objectif  de 
mettre  en  place  des  concepts  nouveaux  ou 
Tutilisation  large  de  tels  fichiers  permettrait  de 
conduire  une  partie  importante  de  la  mission,  en 
particulier  en  vol  TBA,  dans  des  conditions  de 
mauvaise  visibility  (tout  temps  et/ou  nuit)  par  le 
biais  de  presentation  au  pilote  d'imageries 
synthetiques 
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Dans  un  premier  temps,  les  aquipes  de 
DASSAULT  AVIATION  ont  ata  chargees  d'atablir 
les  premieres  bases  d'un  tel  concept. 

5.  LES  ORIGINES 

Avant  de  rentrer  dans  ('explication  des  principes 
retenus  pour  le  concept  APIS  propose,  il  est  bon 
de  revenir  sur  des  etudes  anterieures  dans 
fesquelles  on  trouve  les  fondements  de  la 
demarche  suivie  pour  APIS. 

5.1  PERSEPOLIS  et  le  pilotage  tate  haute 

En  France,  dans  les  ann6es  1970,  un  mouvement 
important  s'est  fait  en  faveur  d  un  principe 
nouveau  de  pilotage  des  aaronefs  baptist 
"pilotage  tate  haute".  On  pourra  en  trouver  une 
bonne  description  dans  les  travaux  de 
M.  KLOPFSTEIN  du  CEV  (voir  notamment 
reference  1). 

L'idae  de  base  a  eta  de  montrer  a  I'achelle  1, 
en  superposition  avec  le  monde  extarieur  le 
vecteur  Vitesse,  I'horizon  et  I'achelle  d'assiette. 

Ce  mouvement  s'est  fait  grace  a  un  triple 
progr6s  technologique  : 

-  I'apparition  des  premiers  capteurs  inertiels  de 
precision  et  en  particulier  des  centrales  a  inertie 
capables  de  fournir  attitudes  et  vitesses  avec 
des  erreurs  limitaes  ; 

-  le  daveloppement  des  calculateurs  aiectro- 
niques  embarqu^s  (dabut  des  techniques 
num6riques)  ; 

-  la  mise  au  point  de  viseurs  a  tube  cathodique, 
qui,  bien  que  monochromes,  permettaient 
d'afficher  a  la  demande  une  grande  variata  de 
symboles  graphiques  ou  alphanum6riques  avec 
des  lois  de  presence-absence  ou  de 
deplacement  tres  peu  contraintes. 

Ces  derniers  constituaient  un  progres  r6el  par 
rapport  aux  viseurs  eiectro-mecaniques  (a 
plaques  et  lampes)  daja  presents  dans  les 
avions  de  combat  pour  les  besoins  de  visee  des 
conduites  de  tir.  D'ailleurs  I'aviation  de  combat 
de  par  le  monde  s'est  tr6s  ais6ment  convertie 
dans  les  annees  qui  suivirent  a  ('utilisation  de 
ces  dispositifs  tete  haute  pour  le  pilotage. 

Plusieurs  propositions  ont  ete  faites  pour 
developper  ces  memes  techniques  sur  avions 
civils.  Daja  le  MERCURE  montrait  la  voie,  avec 
un  HUD  a  diodes  et  chez  DASSAULT,  a 
I'occasion  d'un  programme  d'etudes  designe 
PERSEPOLIS,  on  a  cherche  comment  aller 
au-deia  de  ces  premiers  daveloppements  du 
pilotage  tate  haute. 

PERSEPOLIS  a  ata  mena  en  association  avec 
THOMSON-CSF  sur  soutien  de  la  DGA  et  de  la 
DGAC  Son  orientation  majeure  a  ata  de  retenir 
des  principes  visant  a  retrouver  un  pilotage 


naturel  (utilisant  la  pente  et  la  route),  fonda  sur 
la  liaison  avec  le  monde  extarieur. 

La  figure  1  montre  un  cas  typique  de  la 
symbologie  PERSEPOLIS  en  approche.  On  note 
en  particulier,  en  plus  de  I'horizon  (repare  1)  et 
du  vecteur  vitesse-sol  (repare  3),  I'indication  de 
la  pente  potentielle  (repare  6)  et  le  traca  d'une 
piste  synthatique  (repere  8).  Celle-ci  est  trac6e 
a  partir  des  indications  donn6es  par  I'lLS. 

On  se  pose  correctement  en  faisant  coincider  la 
trajectoire  (vecteur-vitesse)  avec  cette  piste, 
comme  on  I'apprend  en  aaro-club  lors  des 
premieres  letjons  de  vol  a  vue.  On  peut  atre  aida 
en  cela  par  deux  raticules  (voir  repares  5  et  7) 
proposant  un  "guidage"  relatif  au  vecteur  vitesse 
et  a  la  pente  potentielle. 

Ce  sont  des  figurations  de  ce  type  qui  sont 
oparationnelles  sur  les  avions  DASSAULT,  en 
particulier  pour  le  programme  MIRAGE  2000,  et 
maintenant  sur  le  RAFALE. 

Le  principe  'retour  au  naturel"  dont  tamoigne 
cette  damarche  va  dans  la  ligne  de  conduite 
pronae  par  I'approche  "facteurs  humains"  : 
comprendre  I'homme,  chercher  a  utiliser  au 
mieux  ses  compatences,  son  habileta,  ses 
"automatismes". 

Avant  de  quitter  le  monde  des  avions  civils,  on 
mentionnera  qu'il  existe  de  plus  en  plus 
d'avions  civils  dotas  de  viseurs  aiectroniques 
permettant  un  certain  pilotage  t§te  haute  et  que 
le  mouvement  gagne  lentement  du  terrain 
malgra  le  surcoOt  que  cette  option  implique.  Des 
HUD  FLIGHT  DYNAMICS  ont  ata  installs  sur  des 
B  727  ou  le  seront  bient6t  sur  des  B  737-400  et 
des  MD-83.  Et  bien  sOr,  un  HUD  SEXTANT 
AVIONIQUE  aquipe  tous  les  A  320  d'AIR  INTER. 
Comme  I'indique  la  FLIGHT  SAFETY 
FOUNDATION  (voir  rapport  cita  en  rafarence  2), 
cet  investissement  se  ravaie  payant  dans  la 
sacurita  du  vol  et  son  impact  dans  la  relation 
homme-machine  est  tras  favorable,  comme  en 
tamoignent  les  syndicats  de  pilotes  (voir  par 
exemple  article  cita  en  rafarence  3).  Par 
ailleurs,  son  intagration  a  l  aiectronique  de  bord 
est  un  point  d'atude  d'architecture  intaressant 
aborda  dans  la  communication  citae  en 
rafarence  4. 

5.2  L'atude  O.P.E. 

O  P  E  pour  Organisation  du  Poste  d'Equipage 
(d'un  Avion  de  Combat)  est  le  nom  d'un  autre 
Daveloppement  Exploratoire  mena  de  81  a  83 
par  DASSAULT  avec  le  concours  de 
THOMSON-CSF  et  SV2  (CROUZET-SFENA)  sur 
soutien  de  la  DGA  (STTE).  Son  but  atait  la 
definition  d  une  cabine  d'avion  d'armes  a  siage 
inclina,  faisant  appel  aux  technologies  nouvelles 
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d'interface  homme-machine.  Le  programme 
s'appuyait  sur  des  maquettes  et  des  essais  en 
centrifugeuse  du  CEV/LAMAS  (voir  reference  5). 

Plusieurs  kfees  mattresses  ou  concepts 
nouveaux  sont  issues  de  ce  D.E.  et  ont  6fe 
appliquees  en  grande  partie  sur  I'Avion 
Experimental  RAFALE-A.  Certains  de  ces 
principes  ont  aussi  efe  repris  pour  le 
programme  MIRAGE  2000-5. 

Le  pilotage  fete  haute  etant  toujours  une  des 
principales  caracferistiques  des  cockpits 
modernes,  I'O.P.E.  a  conduit  a  d6velopper  la 
technologie  des  viseurs  "holographiques"  (a 
optique  diffractive).  Ceux-ci  procurent  sur  les 
viseurs  classiques  deux  avantages  de  taille  : 

-  le  champ  :  30°  d'ouverture  en  gisement  par 
20°  d'ouverture  en  site  est  un  chiffre  typique  et 
on  a  rrfeme  retenu  plus  sur  RAFALE 

-  les  performances  photorrfetriques  :  k  double 
titre,  le  CTH  en  devenant  tr6s  transparent 
n'alfere  que  faiblement  la  vision  de  I'exferieur, 
et  la  symbologie  est  perdue  dans  toutes  les 
conditions  de  vol  (face  au  soleil,  dos  au  soleil...). 

Mais  I'O.P.E.  a  propose  des  concepts  plus 
originaux,  en  particulier  le  principe  du 
Collimateur  Tfite  Moyenne  (CTM)  :  toutes  les 
informations  k  presenter  au  pilote  n'ont  pas 
besoin  d'etre  pr6senfees  en  fete  haute,  n'etant 
pas  Ifees  au  monde  exferieur  (du  moins  a  sa 
partie  accessible  en  vision  directe).  Mais 
pr6senfees  dans  un  6cran  fete  basse  classique, 
elles  peuvent  se  r6v6ler  difficiles  d'acces  par  un 
coup  d'oeil  rapide.  On  cr6e  done  un  terminal  de 
visualisation,  place  g6om6triquement  tfes  pr6s 
du  CTH  (juste  au-dessous)  et  procurant  des 
images  projefees  k  une  distance  equivalente  k 
celles  du  viseur.  Celles-ci  sont  done  rapidement 
accessibles.  De  plus,  la  collimation  (projection  a 
distance)  les  met  k  I'abri  de  bon  nombre  de 
reflets  parasites. 

Le  premier  CTM  a  vofe  sur  I'Avion  Experimental 
RAFALE-A.  Depuis,  des  6quipements  ont  616 
fabriqu6s  en  serie  dans  le  cadre  de  la 
modernisation  des  SUPER-ETENDARD  et  pour  le 
MIRAGE  2000-5.  Ces  equipements,  mono¬ 
chromes,  sont  particulferement  bien  places  pour 
la  presentation  des  visualisations  issues  des 
capteurs  (pods  optroniques  ou  radars).  Pour  les 
RAFALE  destines  k  equiper  la  MARINE 
NATiONALE  et  L'ARMEE  de  L'AIR  FRANCAISE, 
est  developpe  par  SEXTANT  AVIONIQUE  un  CTM 
polychrome  k  grand  champ  (20°  par  20°)  qui 
servira  a  la  fois  pour  les  capteurs  et  aussi  pour 
la  representation  de  la  situation  tactique  Ce 
projet  est  decrit  dans  la  communication  cifee  en 
reference  6. 


En  outre,  un  grand  nombre  de  principes  de 
realisation  dans  les  logiques  de  presentation  ou 
de  commande  ont  efe  proposes  par  le  groupe 
O.P.E.  Beaucoup  de  ceux-ci  ont  efe 
experimenfes  sur  le  RAFALE-A.  Its  femoignent  fe 
encore  tous  d'un  6tat  d'esprit  de  simplicife  et  de 
naturel. 

Prenons  un  exemple  :  on  connatt  classiquement 
dans  les  cabines  la  commande  faisant  passer 
une  centrale  k  inertie  de  I'etat  "alignement"  k 
I'etat  "navigation".  Avant  de  quitter  le  parking,  la 
consigne  est  basculer  "d'alignement"  a 
"navigation*.  Si  cette  manoeuvre  est  oublfee  et 
qu'on  commence  a  rouler  en  "alignement",  toute 
la  procedure  d'alignement  est  k  reprendre...  et 
de  precieuses  minutes  sont  perdues,  ce  qui  peut 
etre  un  gros  handicap  pour  des  d6collages  sur 
alerte.  Dans  I'O.P.E,  on  a  tout  simplement 
remplacd  cette  consigne  par  une  astuce  faisant 
passer  le  sysfeme  en  "navigation"  au 
desserrement  du  frein  de  pare.  Ce  dispositif  est 
de  base  sur  RAFALE. 

Les  premieres  esquisses  de  terrain  synthetique 
en  fete  haute  et  fete  moyenne  (voir  Figure  2)  ont 
6fe  eiaborees  dans  le  contexte  de  I'etude  O.P.E. 

6.  METHODE  DE  TRAVAIL 

Lfetude  APIS,  conduite  sur  deux  ans,  a 
clairement  efe  separee  en  deux  etapes 
majeures,  chacune  donnant  lieu  k  plusieurs 
presentations  aux  Services  Officiels. 

6.1.  Les  images  statiques 

La  premiere  etape  a  permis  I'analyse  exhaustive 
d'un  tr6s  grand  nombre  de  solutions  de 
representations,  sous  forme  d'images  statiques, 
et  s'est  terminee  par  la  realisation  de  films 
d'animation.  C'est  ainsi  qu'on  a  pu  regarder  en 
fete  haute  par  exemple,  ce  que  donnait,  sur 
differtnts  types  de  terrain  (plat,  vallonne, 
montagneux. ..)  differents  principes  de 
representation  : 

-  carroyage  fixe  au  sol 

-  carroyage  attache  k  I'avion 

-  transversales 

-  fuyantes 

-  lignes  de  cretes 

-  noeuds 

On  a  aussi  regarde  quels  elements  il  fallait 
ajouter  au  paysage  ainsi  construit  pour  aider  k 
son  interpretation.  Un  point  tres  etudfe  a  ete 
celui  des  "arbres".  Dans  la  perception  naturelle, 
la  vue  des  arbres  au  sol  permet  d'apprecier  la 
hauteur  de  vol,  beaucoup  plus  instinctivement 
qu'un  compteur  ou  qu'une  echelle.  On  a  done 
cherche  a  implanter  dans  ces  paysages 
synthetiques  des  arbres  de  differentes  formes  et 
differentes  hauteurs.  L'implantation  de  ces 
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arbres  a  elle-m§me  ete  etudi6e  :  alignements, 
implantations  aieatoires  ou  repartition  reguliere 
le  long  de  la  trajectoire... 

Oans  les  images  tete  moyenne,  c'est  aussi 
('utilisation  de  la  couleur  et  des  differentes 
6cheiles  qui  ont  ete  des  facteurs  importants  de 
parametrage. 

6.2.  Les  films  d'animatlon 

Les  images  de  tete  haute  et  de  tete  moyenne 
qui  semblaient  les  plus  efficaces  ont  donn6  lieu 
£  la  realisation  d'images  animees.  Ces  images 
ont  6t6  obtenues  en  enregistrant  sur  un 
magnetoscope  vue  par  vue  les  images 
calcuiees.  Dans  cette  etape,  le  souci  majeur 
n'etait  pas  celui  de  la  performance  "temps  reel". 
On  a  ainsi  r6ussi  £  cr6er  des  films  de  relief 
synthetique  sur  des  terrains  existants  en  France. 
En  utilisant  en  outre  des  trajectoires  r6elles  sur 
ces  terrains,  ces  films  synthetiques  ont  pu  etre 
compares  £  des  vols  reels  faits  en  suivi  de 
terrain  par  un  MIRAGE  2000-N  dont  on  avait 
enregistre  la  tete  haute. 

Ces  realisations  ont  permis  d'orienter  les 
premiers  choix  et  d'amorcer  la  deuxi£me  etape 
de  I'etude. 

6.3.  Le  travail  en  simulation 

Cette  phase  a  ete  conduite  principalement  au 
centre  de  simulation  OASIS  de  DASSAULT 
AVIATION  £  Istres. 

On  est  done  parti  des  propositions  les  plus 
probantes  faites  £  retape  prec6dente,  avec  cette 
fois,  de  base,  la  preoccupation  du  temps  de 
cafcul. 

Des  essais  preaiables  avaient  eu  lieu,  en 
paralieie  du  travail  precedent  sur  les  images 
fixes,  pour  faire  des  choix  sur  les  architectures 
informatiques  (mat6rielles  et  logicielles)  £ 
retenir  dans  I'objectif  du  temps  reel.  Ces  choix 
ont  ete  presentes  par  M.  Roland  MIGINIAC  au 
congres  AGARD/FMP  de  Bruxelles  en  Octobre 
1991  (voir  reference  7). 

6.4.  La  concerts tlon  avec  les  Services  Officiels 

La  methode  de  travail  retenue  permettait  done 
de  passer  en  revue  un  grand  nombre  de  cas  par 
les  differentes  param£trisations  evoquees.  Le 
contrdle  de  projet  mis  en  place  avec  les 
Services  Officiels  a  permis  de  fixer  les  choix 
gr£ce  £  de  fr£quentes  rencontres  avec  le  BPSA, 
le  CERMA  et  le  STTE.  Ces  presentations  ont 
p£riodiquement  ponctue  les  d6veloppements  ; 
elles  ont  permis  £  chaque  fois  de  dresser  un 
etat  des  lieux,  de  recentrer  ou  de  r6-orienter  les 
recherches  et  de  repartir  quelquefois  dans  de 
nouvelles  directions.  C'est  ainsi  par  exemple 
qu'est  n£e  au  cours  d'une  seance  de  travail 


avec  les  Services  Officiels  I'idee  de  la  video 
pleine  CTH  (HUD4) 

7.  LA  CONSTRUCTION  DU  MONDE  APIS 

Conform6ment  £  Tesprit  d'origine  du  concept,  la 
volonte  d'une  representation  naturelle 
"instinctive"  a  oriente  les  premieres  etudes.  Le 
point  de  depart  6tait  de  representer  le  monde 
environnant  £  l'£chelle  1  dans  le  CTH,  en 
superposition  du  terrain  reel  (lorsqu'il  est 
visible)  ou  en  remplacement  (sinon). 

En  fait,  cette  idee  est  progressivement  apparue 
limitee  du  fait  de  la  relative  faiblesse  du  champ 
CTH  vis-£-vis  de  la  reproduction  des  effets  de 
vision  peripherique.  D'ou  aussi  la  recherche  de 
presentations  en  pianche  de  bord,  ce  qui  permet 
de  s'affranchir  de  I'echelle  1  et  mfeme  de 
prendre  du  recul  (ou  de  I'avance  !)  par  rapport  £ 
la  situation  r£elle. 

7.1.  APIS  en  tete  haute 

L'image  synthetique  dessin6e  dans  le  CTH  doit 
respecter  un  certain  nombre  de  regies 
eiementaires  : 

-  discretion  de  la  symbologie  du  terrain  par 
rapport  aux  reticules  classiques  du  pilotage  tete 
haute  ; 

-  clar16  et  lisibilite  de  la  presentation  ; 

-  transparence  de  I'ensemble  pour  voir  le 
paysage  reel  quand  c'est  possible.  Ces 
considerations  ont  amen6  £  imaginer  differentes 
solutions  qui  ont  suivi  la  filiere  images  statiques, 
films  d'animation  et  enfin  simulation  pour  celles 
qui  apparaissaient  les  plus  prometteuses  (HUD1, 
HUD2  &  HUD3).  L'image  HUD4  a  ete  adoptee  au 
cours  des  travaux  en  simulation. 


Denomination 

HUD1 

HUD2 

HUD3 

HUD4 


Type  de  representation 

Lignes  iso-distances 
Lignes  de  crete 
Carroyage  geographique 
Video  pleine 


La  solution  HUD2.  tr£s  appreci£e  lors  des 
presentations  d'images  statiques,  a  ete  ecartee 
lors  des  essais  en  dynamique  puis  en 
simulation.  En  effet,  les  lignes  de  crete  relient 
des  points  g6ographiques  qui,  pour  une  position 
d'observation  donnee  correspondent  £  un 
maximum  local  du  site  d'observation.  Ainsi,  leur 
presence  et  leur  nombre  depend  beaucoup  du 
relief  du  terrain  et  surtout  de  Taltitude  de  I'avion 
et  de  ses  evolutions.  Les  pilotes  sont  genes  du 
fait  de  leur  densite  constamment  variable  et 
plut6t  impr£visible  ainsi  que  par  les  evolutions 
du  trace  en  fonction  de  Tavancement  de  I'avion. 
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Les  carroyages  correspondent  £  la 
representation  du  relief  par  le  dessin  a  une  grille 
fllaire  £  mailles  carries  plaqu£e  sur  le  sol.  La 
solution  HUD3  utilisant  ces  principes  de 
carroyage,  fixes  au  sol,  a  finalement  £t£  rejet6e 
car  elle  introduisait  une  charge  graphique  trap 
importante,  nuisible  £  la  bonne  vision  des 
reticules  classiques  et  du  terrain  reel,  sans  etre 
plus  6vidente  a  interpreter  que  HUD1. 

Le  principe  des  images  MUD1  est  simple  :  a 
intervalles  reguliers,  on  trace  des  lignes 
perpendiculaires  a  la  route  avion  qui  epousent 
le  relief  du  terrain.  C'est  cette  voie  qui  est 
apparue  la  plus  interessante.  Les  lignes 
iso-distantes  permettent  une  interpretation  facile 
du  terrain  environnant,  tout  en  offrant  une  bonne 
lisibilite  du  reste  de  la  symbologie. 

On  a  pu  encore  faire  progresser  cette 
representation  en  incorporant  un  certain  nombre 
d'amenagements,  tel  que  le  parametrage  de  la 
distance  entre  les  lignes,  reglable  £  la  demande 
par  une  commande  £  la  disposition  du  pilote 
("taux  de  terrain').  Mais  I'astuce  la  plus 
determinante  a  6t6  d'introduire  un  estompage 
progressif  de  la  luminosite  de  ces  lignes  en 
fonction  de  leur  distance  a  I'avion.  Celui-ci  a 
aussi  6t6  teste  avec  HUD2  et  HUD3. 

Cet  estompage  est  interessant  £  plusieurs 
titres  : 

-  il  permet  de  retrouver  le  ph£nom6ne  naturel 
physique,  qui  est  un  des  facteurs  monoculaires 
influant  dans  la  perception  du  relief  (voir 
reference  8) 

-  il  permet  de  lutter  contre  I'affiux  de  Iumi6re 
provoque  par  le  tassement  de  lignes 
correspondant  aux  distances  lointaines 

-  il  permet  d'amortir  I'apparition  brusque  de 
lignes  £  I'horizon,  li£e  £  la  taille  de  la  zone  de 
terrain  num£rique  pris  en  compte  pour  le  calcul 
de  I'image. 

Enfin,  nous  avons  6t6  amen6s  £  consid6rer 
('utilisation  d'une  image  non  plus  filaire  comme 
les  pr£c£dentes  mais  g£n6ree  de  fa$on  similaire 
aux  images  t£te  basse,  sous  forme  de  'video 
pleine'.  Baptist  HUD4,  ce  type  d'image  est 
obtenu  en  g6n£rant  en  chaque  pixel  une  teinte 
verte'*1  plus  ou  moins  lumineuse  en  fonction  des 
ombres  cr££es  par  le  relief  sur  le  terrain.  En 
surimpression  de  cette  image,  qui  pour  certains 
ressemble  beaucoup  £  une  video  d'un  capteur 
optronique,  figurent  toujours  des  reticules 
classiques  de  pilotage.  La  trajectoire  peut  aussi 
etre  repr£sent£e  ;  la  solution  la  meilleure 
semblant  £tre  alors  de  la  dessiner  sous  la  forme 


d'un  ruban  "noir",  c'est  £  dire  en  fait 
transparent. 

Dans  ce  type  d'image,  la  vision  du  monde 
exterieur  par  transparence  est  envisageable  par 
un  r£glage  approprie  de  la  luminosity  g6n£rale. 
D'ailleurs,  cette  image  semble  plus  adaptee  aux 
vols  en  ambiance  cr£pusculaire  qu'aux 
luminances  £levees  du  fait  des  capacites  des 
viseurs  £  gen£rer  de  forts  niveaux  de  lumiere 
tout  en  assurant  un  balayage  video  complet. 

Pour  toutes  ces  figurations  t£te  haute,  I'habillage 
de  I'image  APIS  brute  du  terrain  comprend  : 

-  des  symbologies  de  pilotage,  reprises  des 
symbologies  classiques  utilis£es  sur 
MIRAGE  2000  et  RAFALE  ;  en  complement 
certains  essais  de  reticule  ont  6te  faits  de  type 
directeurs  d'ordres  pour  aider  £  la  reprise  en 
main  ;  des  precautions  particuli£res  ont  et£ 
prises  pour  favoriser  la  lecture  des  reticules  sur 
le  fond  d'image  APIS. 

-  des  indications  relatives  aux  obstacles  et  £  la 
trajectoire  de  reference  ;  cette  trajectoire  est 
celle  que  I'avion  doit  suivre,  telle  qu'elle  resulte 
g6neralement  de  la  preparation  de  mission. 

Pour  les  evaluations  pilotees,  ce  sont  les 
figurations  type  HUD1  et  HUD4  qui  ont  et£ 
retenues. 

7.2.  APIS  en  t6te  moyenne 

De  base,  les  images  £  g£n6rer  en  tfite  moyenne 
ont  et6  conpues  comme  des  images  pleines, 
puisque  il  n'y  avait  plus  £  assurer  de 
"transparence".  L'-s  technologies  adoptees  en 
planche  de  bord  de  nos  avions  etant  toutes 
capables  de  produire  de  larges  palettes  de 
couleurs,  nous  sommes  d'embiee  partis  sur  la 
definition  d'images  statiques  polychromes. 

Tr£s  vite,  il  est  apparu  qu'il  etait  interessant  de 
presenter  des  images  repr6sentant  un  champ 
plus  important  qu'en  CTH.  L'hypoth£se  nominale 
de  ('utilisation  d'un  CTM  polychrome  de  type 
RAFALE  (de  20°  par  20°)  nous  a  tout  d'abord  fait 
envisager  des  images  de  format  carr6  represen- 
tant,  par  exemple,  des  zones  de  60°  par  60°. 

Mais  ce  CTM  etant  capable  de  gerer  plusieurs 
images  en  multi-fen£trage,  il  est  aussi  apparu 
interessant  d'adopter  un  format  d'image  de  10° 
de  hauteur  par  20°  de  largeur.  En  effet,  les 
grandes  ouvertures  angulaires  en  site  sur  le 
terrain  n'apportent  pas  beaucoup  d'aide.  Par 
contre,  ('utilisation  de  I'autre  moitie  du  CTM  peut 
etre  laissee  £  d'autres  fonctions  (gestion  du  vol, 
situation  tactique  horizontale,  conduites  de  tir ..). 


<**  Etre  monochrome  permet  la  compatibility  avec  les  CTH  multimodes  existants 
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Le  travail  sur  images  statiques  a  ygalement 
permis  de  dyfinir  une  politique  d'utilisation  de  la 
couleur.  Apr£s  plusieurs  t£tonnements,  au 
codage  hypsom6trique<3>  a  yty  pr6f6r6  le 
monochrome  (souvent  £  base  grise)  dans  le  but 
de  laisser  libre  ('utilisation  des  couleurs  pour 
d'autres  habillages  soit  de  nature  planimytrique 
(villes,  fleuves,  forSts,  etc)  soit  de  nature 
tactique  (menaces,  zones  de  responsability, 
autres  avions,  etc). 

Mais  il  reste  encore  un  grand  degrd  de  liberty  : 
celui  du  point  d'observation  Naturellement,  on 
peut  calculer  I'image  de  la  position  pr£sente, 
comme  pour  la  tyte  haute.  Toutefois,  notre  ytude 
6tant  centre  sur  le  vol  TBA,  a  yty  vite  6voqu6e 
la  possibility  de  'prendre  de  la  hauteur',  ce  qui 
permet  en  particulier  d'6loigner  I'horizon 
optique.  Dans  ce  contexte,  on  a  ytudiy  une 
'image  d'yiyvation",  vue  depuis  un  point 
d'observation  suffisamment  au-dessus  de  la 
position  avion  pour  que  le  relief  situ6  £  moins 
d'une  distance  donnye  (15  km,  typiquement)  ne 
soit  pas  cachy  par  un  masque  de  terrain. 

Passy  ce  premier  stade,  on  a  aussi  pensy  a 
"prendre  du  recul',  ce  qui  permet  de  mieux  voir 
la  situation  de  part  et  d'autre  de  I'avion,  oil 
peuvent  se  trouver  des  yquipiers  par  exemple. 

A  I'inverse,  on  a  aussi  pensy  a  prendre  de 
I'avance  et  ont  yty  simuiyes  des  "vues 
d'anticipation',  vues  fixes  ou  recalcuiyes  en 
faisant  bouger  le  point  d'observation  ou  t  angle 
de  vue  selon  les  souhaits  du  pilote.  L'idye  ytait 
de  visualiser  ainsi  I'arrivye  sur  un  objectif  ou  un 
passage  dyiicat  de  I'itinyraire,  plusieurs  minutes 
a  I'avance,  avec  la  possibility  de  zoomer,  de 
s'yiever,  de  changer  d'axe,  etc.  Toutefois,  ces 
images  ont  ensuite  yty  ycartyes  car  elles 
semblent  relever  plus  de  la  preparation  de 
mission  que  du  vol  TBA  dans  un  avion 
monoplace. 

C'est  en  fait  I'addition  de  diffyrents 
dyplacements  du  point  d'observation  qui  a 
abouti  au  meilleur  compromis,  avec  la  cryation 
de  I'image  "demi-dieu".  Cette  dynomination141  a 
yty  adoptye  car  I'image  reprysente  un  moyen 
terme  entre  la  situation  horizontal  bidimension- 
nelle  avec  son  habillage  cartographique  ("vue  de 
Dieu'  ou  'God's  eye')  et  la  vue  directe  du 
cockpit  Dans  la  vue  demi-dieu,  on  regarde  le 
terrain  environnant  £  partir  d’une  position 
recuiye  et  yievye  par  rapport  £  la  position 
ryelle.  Dans  cette  image,  pyriodiquement 
recalcuiye,  mais  qui  n'yvolue  ni  en  roulis  ni  en 


tangage  est  representye  la  position  ryelle  par 
une  "maquette"  de  I'avion,  dont  les  attitudes 
varient. 

L'habillage  le  plus  souvent  retenu  est  constitue 
par  : 

-  la  trajectoire  de  ryfyrence  de  I'avion  (mSme 
dyfinition  qu'en  tete  haute),  plaquye  au  sol 

-  des  yiyments  de  planimetrie  :  for§ts,  villes, 
fleuves... 

-  des  obstacles  ou  dangers  divers  (lignes 
haute-tension,  eg) 

-  les  menaces  sol-air,  pour  lesquelles  on  a 
choisi  de  faire  figurer  en  couleur  rouge-orangy, 
des  zones  de  danger  ;  celles-ci  sont  obtenues 
par  projection  au  sol  de  I'intervisibility  entre  la 
menace  et  I'avion  £  la  hauteur  de  vol  courante. 

Comme  yvoqu£  pour  la  tyte  haute,  on  a 
ygalement  cherchy  £  incorporer  £  ces  images 
un  estompage  progressif  de  lumiyre  qui  se 
traduit  ici  par  un  effet  de  brume  dans  les 
tointains. 

Apres  differents  (et  laborieux)  ryglages  sur  les 
diffyrents  paramytres  en  particulier  de  recul, 
d'yievation,  de  brume,  etc,  c’est  ce  type  d'image 
qui  a  yty  prysenty  aux  yvaluations  pilotyes. 

8.  EVALUATION 

A  la  fin  de  I'ytude,  ('organisation  de  Evaluation 
APIS  en  simulation  pilotye  £  OASIS  a  yty 
prepare  et  dyfinie  avec  le  CERMA,  puis 
approuvye  dans  son  principe  par  les  Services 
Officiels,  qui  ont  d6signy  dix  pilotes  evaluateurs. 
Ceux-ci  etaient  aussi  bien  des  pilotes  d'essais 
que  des  pilotes  des  Forces. 

8.1.  Methodes  et  scdnarios 

La  charge  globale  de  travail  demandye  £ 
chaque  pilote  ytait  importante  pour  : 

-  se  familiariser  avec  la  cabine  RAFALE,  ses 
commandes  et  I'environnement  de  simulation 

-  assimiler  et  yvaluer  les  differentes  figurations 
APIS  en  t£te  haute  et  en  tete  moyenne,  sur  trois 
zones  de  relief  diffyrents 

Chaque  yvaluation  a  de  ce  fait  yty  scindye  en 
deux  syances  distinctes  (de  pryparation,  puis 
devaluation)  syparyes  de  quelques  jours 
d'assimilalion 

Comme  il  se  doit,  les  pilotes  evaluateurs 
n'avaient  pas  auparavant  participe  aux 
simulations  APIS  dans  la  phase  dytude  et  tous 
avaient  une  bonne  expyrience  opyrationnelle  du 
vol  TBA. 


Codage  pour  lequel  on  utilise  diffyrentes  couleurs  selon  laltitude 
<4>  Clin  d'ceil  a  la  mythologie  grecque,  alors  que  le  nom  APIS  appartient  H  la  mythologie  fegyptienne 
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Pour  ces  tests,  la  presentation  des  images  APIS 
etait  disponible  en  tete  haute  et  en  tete 
moyenne.  Le  monde  exteriaur  de  simulation 
etait  coherent  avec  I'image  APIS  en  tete  haute  ; 
mais  un  decalage  de  position  pouvait  etre 
introduit  pour  simuler  une  derive  lente  de  la 
localisation  de  I'avion. 

La  premiere  s6ance  a  permis  generalement  de 
presenter  le  cadre  des  travaux,  les  differentes 
figurations  etudiees  en  pr6cisant  les  figurations 
d  evaluer  (HUD1,  HUD4  et  vue  demi-dieu)  avec 
leurs  differentes  options  (arbres,  obstacles, 
trajectoire,  brume,  etc).  Ces  demonstrations 
permettaient  aussi  de  faire  connaissance  avec 
les  differents  terrains  disponibles  en  simulation 

La  deuxieme  seance,  constituant  revaluation 
proprement  dite,  se  deroulait  selon  le 
programme  suivant  : 

a) PHASE  A  :  Trois  passages  successifs  sur 
une  zone  accidentee  permettent  de  voir  : 

-  une  version  de  reference  sans  image  APIS, 

-  un  parcours  identique  avec  APIS  en  tete  haute 
ou  le  pilote  peut  regler  a  sa  convenance  le  "taux 
de  terrain"  et  le  type  de  figuration  (HUD1  ou 
HUD4)  selon  les  conditions  de  visibilite  ; 

-  un  dernier  parcours  efTectu6  avec  HUD1  ou 
HUD4  et  la  vue  demi-dieu  en  CTM,  en  conditions 
IMC  totales.  Les  reprises  en  main  du  pilotage 
sont  possibles  pour  s'£carter  de  la  trajectoire  de 
reference. 

Les  mfimes  passages  sont  proposes  ensuite  sur 
un  second  terrain  de  relief  different. 

b) PHASE  8  :  Un  troisfeme  terrain  est  utilise 
pour  deux  passages.  Le  mode  de  pilotage 
automatique  est  disponible  mais  des  reprises  en 
main  sont  n£cessaires  pour  6viter  des  menaces 
ou  des  obstacles  imprevus.  Lors  d'un  arret 
inopine  de  la  simulation,  on  demande  au  pilote 
de  se  situer  g£ographiquement. 

c) PHASE  C  :  Le  pilote  peut  demander  avec  une 
totale  liberfe  de  choix  (terrain,  images, 
options  )  de  rejouer  les  differentes  possibilites 
offertes 

8.2.  R4suitats 

Les  r£sultats  de  revaluation  sont  resumes  dans 
le  tableau  presents  en  figure  8  Globalement 
ces  rSsultats  sont  encourageants  et  permettent 
de  penser  que  le  dSveloppement  de  telles 
figurations  peut  apporter  un  avantage  au  pilote 
opSrationnel  en  mission  de  penetration  £  tr£s 
basse  altitude.  De  nombreuses  observations  ont 
bien  sOr  6fe  formufees  qui  permettront  de 
prSciser  ou  de  reprendre  certains  points  (voir 
paragraphe  9.1). 

II  apparait  clairement  du  tableau  quo  I'image 
APIS  tete  haute  permet  une  aide  dans  la  gestion 
de  la  trajectoire  £  court  terme.  Elle  autorise  une 


certaine  reprise  en  main  et  surtout,  elle  permet 
une  meilleure  comprehension  des  actions  des 
automatismes  de  gestion  de  la  trajectoire.  Les 
commentaires  faits  ont  Sgalement  permis  de 
constater  que  I'image  HUD4  a  6t6  jugSe  plus 
facile  £  interpreter  que  la  symbologie  HUD1. 

En  tete  moyenne,  APIS  offre  un  support  possible 
pour  la  situation  tactique. 

9.  DISCUSSION 

9.1.  Corrections  ou  ameliorations  de  simulation 

Les  figurations  presentees  ont  b6n6fici£  d'un 
grand  nombre  d'heures  de  travail,  li£es  en 
particulier  aux  logiciels.  Mais  les  maferiels 
informatiques  progressant  sans  cesse,  on  sait 
que  la  meme  etude,  partant  des  m§mes  bases, 
prendrait  aujourd'hui  un  certain  nombre 
d'orientations  differentes  touchant  non  aux 
principes  du  concept  mais  £  certains  choix  de 
realisation.  Certains  de  ces  choix,  faits  dans  la 
quete  de  la  performance  temps  reel,  pourraient 
etre  revus,  ce  qui  corrigerait  des  critiques 
adressees  lors  de  revaluation.  Par  exemple, 
ceci  concerne  les  trajectoires  dont  le  calcul 
d'6limination  des  parties  cactfees  est  simpliste, 
ce  qui  met  leur  projection  au  sol  quelquefois  en 
conflit  avec  la  representation  du  terrain.  Dans  la 
mftme  idee,  t'6conomie  d'un  traitement  de 
lissage  (anti-aliasing)  en  tete  haute,  donne  des 
images  d£sagr6ables  lors  de  virages  depassant 
20  £  30  degr£s  d'inclinaison  ;  cette  £conomie  ne 
serait  plus  de  mise  avec  un  materiel  de  derni£re 
generation. 

9.2.  Evolutions  du  concept 

Les  tout  premiers  essais  de  paysage  synthetique 
avaient  montre  que  la  vision  du  paysage  reduite 
au  champ  du  viseur  ne  permettait  pas  d'esp6rer 
un  pilotage  manuel  des  images  APIS.  L'etude 
d£crite  ici  s'est  done  orient£e  vers  un  support 
au  pilotage  automatique  avec  des  capacifes  de 
reprise  en  main 

Apres  les  evaluations,  cette  position  apparait  un 
peu  reductrice,  certains  evaluateurs  etant  d'avis 
qu'au  prix  d'ameiiorations  de  presentation  qu  its 
jugent  simples,  les  images  CTH  pourraient 
permettre  une  plus  grande  capacife  de  pilotage 
manuel.  Ceci  toutefois,  doit  etre  examine  avec 
prudence,  compte-tenu  des  risques  inherents 
aux  erreurs  de  fichiers  ou  aux  mauvaises 
localisations. 

9.3.  Embarquabilite 

Des  contraintes  ont  et6  prises  de  base  dans 
l'etude  (champ  des  collimateurs,  en  particulier) 
pour  pouvoir  envisager  une  application 
embarquee  £  moyen  terme  Toutefois,  pour  faire 
des  images  animees  avec  une  cadence 
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suffisante,  il  nous  a  fall u  utiliser  des  moyens 
informatiques  so!  qu'il  est  encore  un  peu 
prgsomptueux  de  considerer  embarquables  avec 
des  objectifs  raisonnables  de  volume  et  de  prix. 

Mais  la  principale  difficult^  viendra  des  aspects 
Ii6s  £  la  security  du  vol.  Les  erreurs  faites  tors 
de  la  constitution  des  fichiers  ou  lors  de  leurs 
manipulation,  les  obstacles  apparus  recemment, 
les  d£calages  entre  le  fichier  et  la  position  reelle 
du  fait  d'une  mauvaise  localisation...  sont  autant 
de  pi£ges  pour  le  vol  au  moyen  d'une  technique 
de  type  APIS. 

10.  SUITES  POTENTIELLES 

C'est  pourquoi,  une  des  suites  les  plus 
interessantes  de  I'etude  est  d'examiner  le 
couplage  de  cette  technique  avec  la  detection  du 
relief  par  des  capteurs.  C'est  surtout  avec  des 
capteurs  6lectro-optiques  (camera  thermique, 
notamment)  que  ce  couplage  apparait  le  plus 
porteur  du  fait  de  la  complementarity  des  deux 
moyens.  Un  contrat  sur  ce  theme  est  en 
preparation,  pour  lequel  seront  utilis6es  les 
images  prises  lors  de  vols  de  la  nacelle  RUBIS 
de  DASSAULT  AVIATION  sur  MIRAGE  2000. 

Le  STTE  a  dgalement  pr6vu  des  essais  en  vol 
sur  avion  banc  d'essai  FALCON  20,  equipe  de 
terminaux  de  visualisation  proches  de  ceux  du 
RAFALE.  Ces  essais  seront  prepares  par  des 
campagnes  de  simulation  au  centre  de 
simulation  du  CEV. 

L'ensemble  de  ces  travaux  permettra  d'affiner  le 
concept  en  vue  d'equiper  par  la  suite  des  avions 
d'armes.  L'application  aux  avions  civils  peut 
ygatement  6tre  envisag6e,  en  particulier  pour 
I'aide  au  vol  au  voisinage  du  sol  lors  des 
operations  de  decollage  et  atterrissage. 

11.  CONCLUSIONS 

Une  etude  de  deux  ans  a  ete  menee  par 
DASSAULT  AVIATION  sur  la  presentation 
d'images  synthetiques  representant  I'environ- 
nement  d'un  avion  en  penetration  TBA.  Une 
phase  de  travail  sur  images  statiques  a  permis 
de  passer  en  revue  de  nombreuses  propositions. 
Les  plus  pertinentes  ont  donn6  lieu  a  des  Films 
d'animation.  Dans  une  deuxieme  phase, 
('utilisation  de  moyens  de  simulation  temps  reel 
a  autorise  la  mise  au  point  de  representations 
en  tete  haute  et  en  t§te  moyenne  Des  pilotes  ont 
pu  6valuer  ces  representations.  Leurs  avis, 
largement  positifs,  militent  pour  une  poursuite 
des  travaux  sur  ce  theme.  Ceux-ci  doivent 
maintenant  porter  sur  fa  question  incontournable 
de  la  security  du  vol  qui  peut  etre  resolue  en 
particulier  en  associant  les  fichiers  de  terrain 
aux  informations  fournies  par  des  capteurs 
embarques. 
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figure  5.  APIS  HUD3 


figure  6.  APIS  HUD4 


figure  7.  APIS  Vue  demi-dieu 

N.B.  :  La  quality  des  photographies  et  de  la  reproduction  ne  peut  pas  rendre  I'impression  des 
images  APIS  obtenues  sur  les  6crans  de  visualisation  (dynamique  de  lumi^re,  ombrages, 
contrastes...) 


EVALUATION  APIS  EN  TETE  HAUTE  PAR  10  PILOTES  MILITAIRES 


Utilisation  fonctionnelle 

Role 

essentiel 

Utile 

Comprehension  de  la  situation 
instantanee 

10 

0 

Anticipation  des  actions  du  pilotage 
automatique 

5 

3 

Pilotage  manuei 

1 

5 

Sans 

interet 


EVALUATION  APIS  EN  TETE  MOYENNE  PAR  10  PILOTES  MILITAIRES 

Utilisation  fonctionnelle 

Role 

essentiel 

Utile 

Sans 

interet 

Comprehension  de  la  situation 
court/moyen  terme 

5 

5 

0 

Anticipation  des  actions  du  pilotage 
automatique 


Pilotage  manuei 


Comprehension  de  la  situation  tactique 


figure  8.  Tableau  de  eynthEse  de#  Evaluation# 
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1.  SUMMARY. 

Successful  applications  of  holographic  optical 
elements  (HOEs)  in  the  holographic  night  vision 
goggles  have  led  to  interesting  developments  of 
new  concepts  for  helmet  mounted  vision  systems. 

The  present  application  of  HOEs  in  night  vision 
goggles  is  discussed,  as  well  as  a  new  concept  of 
Helmet  Mounted  Vision  Systems. 


2.  INTRODUCTION. 

Holographic  Optical  Elements  (HOEs)  have  found 
applications  in  night  vision  goggles  and  helmet 
mounted  night  vision  systems  as  a  consequence  of 
following  important  advantages. 

HOEs  can  be  made  highly  reflective  for  the  light 
emerging  from  the  PT  (image  intensifier  tube) 
without  significantly  decreasing  the  light 
transmission  from  the  outside  world. 

HOEs  can  perform  optical  functions  which  cannot 
be  obtained  with  classical  optical  components. 

The  optical  function  of  the  HOE  can  be  made 
independent  of  the  shape  of  the  substrate  on  which 
it  is  manufactured. 

HOEs  can  be  manufactured  on  various  substrate 
materials  (glass-  plastics).  The  weight  and  the 
safety  of  night  vision  systems  can  be  improved  by 
the  use  of  plastics  instead  of  glass  substrate 
materials. 


In  the  following  the  advantages  of  the  use  of 
holographic  optics  are  demonstrated  in  the  existing 
holographic  night  vision  goggles,  and  future 
developments  of  helmet  mounted  systems  based  on 
a  holographic  visor  are  discussed. 

3.  USE  OF  HOEs  IN  THE  HOLOGRAPHIC 
NIGHT  VISION  GOGGLES. 


3.1.  Working  principle. 

The  working  principle  of  the  night  vision  goggles 
(HNVG)  is  shown  in  fig.  1. 

The  image  of  the  environment  is  projected  onto 
the  entrance  plane  of  the  PT  by  the  objective.  The 
amplified  image  is  projected  to  the  users  eyes  by 
the  relay  optics,  the  flat  and  the  curved  HOEs. 

The  HOEs  have  a  dual  function.  The  optical  rays 
are  deflected  to  the  users  eyes,  and  the  HOEs 
perform  the  same  function  as  a  eye  piece  (a  virtual 
image  is  formed  at  a  large  distance  in  front  of  the 
eyes). 

The  image  intensifier  tube  (I^T)  contains  a 
phosphor  with  a  small  bandwidth.  The  HOEs  only 
have  to  reflect  the  light  centralised  around  the 
peak  wavelength  of  the  phosphor  (543nm)  and  can 
have  a  small  spectral  bandwidth.  A  typical  spectral 
curve  of  both  types  of  HOEs,  recorded  in  DCG 
(dichromated  gelatin),  is  shown  in  fig.2.  The  lower 
efficiency  of  the  flat  HOE  for  p-  polarised  light  is 
caused  by  polarisation  effects  at  the  large  angle  of 
incidence  (53°). 


Fig.  1  :  Working  principle  of  the  HNVG 

Main  elements  are  :  1  -  the  objective 

2  -  the  FT 

3  -  the  relay  optics  :  collimator,  roof  mirror,  relay  lenses,  mirror 

4  -  curved  and  flat  HOEs 


Rg  2b  wavelength  (nm) 

Fig.  2  :  Transmission  of  the  HOEs  as  a  function  of  the  wavelength 
2a  :  curved  HOE,  angle  of  incidence  8°,  unpolarized  light 
2b  :  flat  HOE,  angle  of  incidence  53°,  s  and  p  polarized  light 


The  HOEs  have  to  be  made  in  such  a  way  that  all 
the  rays  generated  by  the  FT  and  going  through 
the  optical  system  are  reflected  to  the  users  eyes 
with  the  same  high  efficiency.  To  guarantee  an 
almost  constant  diffraction  efficiency  over  the  total 
field  of  view,  a  point  source  located  at  the  centre 
of  the  exit  pupil  of  the  optical  system  is  used  to 
record  the  HOEs.  During  replay,  all  field  rays 
arriving  at  this  exit  pupil  position  are  reflected 
with  the  same  high  efficiency. 

To  obtain  a  high  diffraction  efficiency  at  other 
pupil  positions  as  well,  the  HOEs  need  to  have  a 
finite  angular  and  wavelength  bandwidth.  Light 
rays  arriving  at  the  edges  of  the  exit  pupil  are 
diffracted  by  the  HOEs  at  a  different  angle  than 
rays  arriving  at  the  centre.  A  change  of  the  angle 
of  incidence  corresponds,  according  to  the  Bragg 
equation,  to  a  change  of  the  reconstruction 
wavelength.  This  wavelength  shift  is  especially 
important  for  the  flat  HOEs: 


The  relation  between  the  angle  of  incidence  at  a 
HOE  and  the  peak  reconstruction  wavelength  can 
be  derived  by  differentiating  the  Bragg  condition 
ref.  [1]  : 

£  -  JLsin  »  -  9)  (1) 

dA,  4ttn 

with  :  0  =  the  angle  of  incidence  in  DCG 

4>  =  the  angle  between  the  grating  vector 
and  the  normal  to  the  HOE  substrate  (= 
90°,  conformal  HOE  case) 

K  =  the  grating  vector 
n  =  the  index  of  refraction  of  DCG 
\  =  the  peak  reconstruction  wavelength 

At  small  angles  of  incidence  (<10°,  curved  HOEs) 
a  change  of  the  angle  of  incidence  at  the  HOE 
hardly  influences  the  peak  reconstruction 
wavelength.  But  at  large  angles  of  incidence  (53° 
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for  the  central  rays  of  the  flat  hologram,  fig.3)  a 
small  change  of  the  angle  of  incidence  shifts  the 
peak  wavelength  significantly.  At  a  given  location 
at  the  HOE,  the  difference  between  the  angle  of 
incidence  of  rays  diffracted  to  the  centre  of  the 
exit  pupil  and  to  the  edges  is  about  6°.  To 
guarantee  efficient  reflection  of  all  the  rays 
generated  by  the  PT  and  going  through  the  optical 
system,  the  bandwidth  of  the  flat  HOE  has  to  be 
at  least  32  nm. 


Fig.  3 :  Relation  between  the  reconstruction 
wavelength  and  the  angle  of  incidence. 
This  relation  is  derived  form  the  Bragg 
equation.  The  HOE  is  designed  to  diffract 
rays  at  53°  angle  of  incidence,  and 
wavelength  543 nm. 


The  HOEs  are  thermally  tuned  and  stabilised  at 
the  peak  wavelength  of  the  PT  and  sealed  between 
two  glass  substrates. 


3.2.  Photopic  transmission. 

The  see  through  capability  of  the  HOEs  can  be 
calculated  in  the  following  way: 

Environmental  light  enters  the  flat  HOE  at  an 
angle  of  incidence  which  is  different  from  the 
angle  of  incidence  of  the  night  image  rays. 
Therefore,  according  to  the  Bragg  equation,  the 
spectral  curve  of  the  flat  HOE  is  shifted,  as  shown 
in  fig.4.  To  calculate  the  photopic  transmission, 
the  wavelength  dependence  of  the  eye  sensitivity 
has  to  be  taken  into  account.  The  photopic 
transmission  of  the  HOEs  is  obtained  by 
multiplying  the  transmission  of  the  HOEs  with  the 
eye  sensitivity  curves.  The  photopic  transmission 


is  higher  than  40  %;  a  slightly  pink  discoloration 
of  the  outside  world  is  observable. 


Fig.  4:  Transmission  of  environmental  light  rays 
through  the  curved  and  the  flat  HOE.  The 
reconstruction  wavelength  of  the  flat  HOE 
is  shifted  to  higher  wavelengths,  the 
environmental  rays  enter  at  the  HOE  at  a 
smaller  angle  of  incidence  than  the  night 
image  rays. 

The  advantage  of  the  use  of  HOEs  in  front  of  the 
users  eyes  is  obvious:  The  user  never  looses  the 
image  of  the  outside  world,  even  when  a  sudden 
intens  illumination  of  the  environment  takes  place. 

3.3.  Future  developments  on  the  holographic 
night  vision  goggles. 

Following  improvements  may  be  applied  to  the 
existing  HNVG: 

A  first  improvement  can  be  obtained  by 
manufacturing  the  HOEs  on  the  plastic  housing  of 
the  HNVG  (polycarbonate  visor  and  backplate). 
Since  the  radius  of  curvature,  position  and  angular 
orientation  of  the  visor  is  different  from  the  curved 
glass  substrates,  a  slight  redesign  of  the  optical 
system  is  necessary.  Furthermore,  problems  with 
the  use  of  plastics,  which  are  discussed  in  more 
detail  in  the  next  paragraph,  have  to  be  taken  into 
account. 

A  second  improvement  can  be  obtained  by 
replacing  the  curved  and  flat  HOE  by  one 
hologram.  By  this  way  the  weight  of  the  optical 
system  is  decreased  and  the  discoloration  of  the 
outside  world  is  further  reduced. 


4.  DEVELOPMENTS  OF  A  NEW  CONCEPT 
OF  THE  HELMET  MOUNTED  VISION 
SYSTEMS. 

4.1.  Operational  requirements. 

The  progress  of  the  technology  used  in  the 
holographic  night  vision  goggles  can  result  in  the 
realization  of  a  Helmet  Mounted  Vision  System 
(HMVS)  with  a  holographic  visor  for  a  pilot  in  a 
helicopter  or  fixed  wing  aircraft 

The  operational  requirements  for  such  a  HMVS 
can  be  summarized  as  follows: 

-  Optimization  for  the  following  tasks: 

Air-fight,  air-attack,  air-defence,  tactical 
air-reconnaissance,  support  of  other 
air-operations. 

-  Safety  and  environmental  requirements  similar  to 
those  for  a  conventional  pilot  helmet 


-  A  display  of  symbology  and  night  vision  images 
with  sufficient  brightness  and  quality, 
superimposed  on  the  pilot’s  view  of  the  outside 
world. 

-  Preferably  protection  against  laser  radiation  and 
nuclear  flashes. 

-  Center  of  gravity  and  weight  compatible  with 
human  factor  requirements,  in  particular  in 
high-g  conditions  for  fixed  wing  applications. 

-  Adequate  life  support  and  communication 
system. 

-  Accurate  head  position  sensor  system. 

From  the  technical  point  of  view  the  realization  of 
a  HMVS  which  complies  with  the  above 
mentioned  requirements  is  non-trivial,  in  particular 
with  regard  to  weight,  image  quality  and  safety. 
A  system  concept  will  be  discussed  in  the  next 


Fig.  5  :  Basic  concept  of  an  integrated  night  vision  helmet  (HMVS) 


sections,  where  emphasis  is  given  to  a  new  type  of 
display  subsystem, which  is  one  of  the  key 
elements  of  the  concept 

Present  display-subsystems  are  characterized  by 
limitations  in  peripheral  view  and/or  field  of  view, 
transmission,  weight  and  safety.  The  use  of  a 
subsystem  based  on  a  holographic  visor,  will 
significantly  diminish  these  limitations. 

4.2.  System  concept. 

A  basic  concept  of  an  integrated  night  vision 
helmet  (Helmet  Mounted  Vision  System,  HMVS) 
is  shown  in  fig.  5. 

The  system  basically  consists  of  a  helmet  with 
either  bi-ocular  or  monocular  display  system: 

-  Display  subsystem  consisting  of: 

CRT,  Relay  optics.  Holographic  visor 

-  Low  Light  Level  Television  Sensor  (inside  or 
outside  helmet) 

-  Head  position  sensor 

-  Communication  system 

The  concept  has  a  high  degree  of  flexibility. 


Outside  the  helmet,  flight  data  can  be  combined 
with  FLIR  (Forward  Looking  Infrared  =  Thermal 
Infrared)  and/or  LLLTV  images,  resulting  in  a 
(processed)  video  signal  that  is  presented  to  a 
miniature  CRT. 

The  CRT  image  is  presented  to  the  pilot  by  an 
optical  relay  system  and  holographic  visor.  The 
result  is,  that  for  the  pilot,  the  CRT  image  is 
superimposed  on  the  image  of  the  outside  world 
that  is  transmitted  by  the  visor. 

Whereas  several  helmet  concepts  show  optical 
parts  between  the  pilot’s  eyes  and  the  visor  (ref. 
[2],  [3]),  our  concept  is  free  of  any  optical  element 
between  the  front  head  and  the  holographic  visor 
ref.  [4]. 

Nowadays,  apart  from  the  HOE  visor  concept,  two 
other  categories  of  display  subsystems  concepts 
can  be  distinguished: 

-  Concepts  with  combiner  display  optics  between 
the  pilot’s  eye(s)  and  the  visor. 

-  Concepts  with  a  visor  that  acts  as  a  small 
bandwidth  mirror  for  the  CRT  image  projected 
by  relay  optics. 

The  advantages  of  the  HOE  concept  as  compared 
to  the  other  two  types  are  shown  in  table  1. 


Type. 

Main  feature  of  non- 
holographic  concept. 

Advantage  of  HMVS 
holographic  display  concept. 

Combiner  display. 

Optical  elements  in  front  of  the 
eye. 

No  optical  elements  in  front  of 
the  eye,  no  safety  risks. 

A  better  eye  relief. 

Lower  weight. 

Conventional  visor  projection. 

Non-holographic  visor,  no 
optical  elements  in  front  of  the 
eye. 

Better  reflection  and 
transmission  of  the  visor. 

Larger  field  of  view. 

Table  1  :  Advantages  of  the  holographic  display  concept  as  compared  to  conventional  visor  projection 
systems  and  systems  with  optical  elements  between  the  visor  and  the  eyes. 
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4.3.  Holographic  visor  concept. 

In  looking  for  a  possibility  to  integrate  a  optical 
projection  system  (projection  of  a  night  image  and/ 
or  symbology)  in  the  helmet  of  a  pilot,  without  the 
need  to  insert  optical  components  in  front  of  the 
pilots  eyes,  integration  of  an  holographic  optical 
element  in  the  visor  of  the  helmet  is  a  possible 
solution.  During  the  design  of  the  system 
difficulties  arising  from  the  optical  function  of  the 
HOE  and  from  the  material  properties  of  the 
polycarbonate  visor  have  to  be  taken  into  account. 

4.3.1.  Optical  function  of  the  HOE. 

During  the  design  of  the  optical  system  one  has  to 
aim  at  a  symmetrical  HOE  configuratioa  In  such 
a  system  the  angle  of  incidence  at  the  HOE  is 
equal  to  the  angle  of  diffraction,  or  the  HOE  acts 
like  a  mirror.  Furthermore,  one  has  to  aim  at  an 
off  axis  angle  which  is  as  small  as  possible  for 
following  reasons. 

During  recording  of  a  HOE,  a  periodic  modulation 
of  the  index  of  refraction  is  created.  Surfaces  of 
equal  index,  called  fringe  planes,  are  formed.  If 
the  HOE  is  used  in  a  symmetrical  configuration, 
the  fringe  planes  are  parallel  to  the  hologram 
surface.  In  asymmetric  HOEs,  or  HOEs  with 
slanted  fringes,  the  fringe  planes  intercept  the 
HOE  surface.  As  follows  from  the  grating 
equation,  asymmetric  HOEs  suffer  from  a  large 
amount  of  chromatic  aberrations,  even  when  a 
phosphor  with  a  narrow  bandwidth  is  used. 

The  variation  of  the  angle  of  diffraction  i’  with  the 
wavelength  X  ,  or  the  chromatic  aberration  of  a 
reflection  hologram,  is  calculated  from  the  grating 
equation: 

K  =  _ h _ _  (2) 

sin  i(X)  +  sin  i  '(X) 

with  :  i  =  the  angle  of  incidence 
i’=  the  angle  of  diffraction 
K  =  a  constant 
X  =  the  wavelength 

The  difference  between  the  angle  of  diffraction  at 
two  different  wavelengths  Xq  and  X,  at  the  same 
angle  of  incidence  i,  or  the  dispersion  of  the  HOE, 
is  calculated  from  : 

stal'd,,.^! (3) 
The  HOE  is  free  from  dispersion  if  i’(Ao)  =  FIX,). 


This  condition  is  fulfilled  if  i’(Xo)  =  -  i,  or  if  the 
HOE  is  symmetrical. 

Monochromatic  aberrations,  appearing  if  the  light 
enters  the  hologram  during  replay  in  a  different 
way  as  during  recording,  also  increase 
dramatically  if  the  HOE  is  used  in  a  asymmetrical 
configuration.  These  aberrations  are  also  strongly 
dependent  on  the  off-  axis  angle  at  which  the  HOE 
is  used. 

Another  problem  related  to  HOEs  with  slanted 
fringes,  is  called  flare.  At  the  HOE  surface, 
modulations  in  the  index  of  refraction  of  DCG 
and/  or  modulations  in  the  thickness  of  the 
holographic  emulsion  by  swelling  or  shrinkage 
after  exposure  can  create  a  thin  transmission 
hologram.  Diffraction  of  bright  objects  by  this 
extra  transmission  hologram,  generating  spurious 
rainbow  patterns,  can  be  very  disturbing  for  the 
pilot 

4.3.2.  Material  properties  of  polycarbonate. 

Other  difficulties  related  to  the  integration  of  a 
HOE  in  the  visor  of  a  helmet  are  caused  by  the 
material  properties  of  the  polycarbonate  visor 
itself.  A  first  problem  is  related  to  sealing  of  the 
HOE  against  humidity.  Polycarbonate  is  not 
waterproof  and  HOEs  in  DCG  are  not  resistant  to 
humidity.  If  humidity  reaches  the  hologram,  the 
peak  reconstruction  wavelength  i.e.  the  wavelength 
for  which  the  Bragg  condition  is  fulfilled  at  a 
given  angle  of  incidence,  shifts  to  higher  values 
and  the  diffraction  efficiency  decreases.  Therefore, 
special  sealing  precautions  have  to  be  taken  to 
prevent  humidity  to  reach  the  hologram. 

A  further  problem  is  generated  by  internal  stresses 
in  the  polycarbonate  visor,  which  introduce 
optical  birefringence.  Optical  birefringence  can 
cause  inhomogeneities  in  the  diffraction  efficiency. 
This  problem  is  discussed  in  detail  in  ref.  [5] 

4.3.3.  Optical  concept. 

As  mentioned  in  paragraph  4.3.1.,  during  the 
design  of  the  optical  system  one  has  to  aim  at  a 
minimum  deviation  of  the  symmetrical  function  of 
the  visor  HOEs. 

The  HOEs  on  the  visor  have  to  perform  a  dual 
function  (fig.5).  Rays  from  the  relay  optics  have  to 
be  deflected  to  the  pilots  eyes,  and  the  HOEs  have 
to  perform  the  same  function  as  a  eye  piece. 
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To  meet  the  above  mentioned  requirements,  one 
has  to  ensure  that  the  difference  between  the  angle 
of  incidence  and  diffraction  at  the  HOE  is  as  small 
as  possible. 

In  the  configuration  shown  in  fig.6,  a  large 
asymmetry  is  introduced  at  the  HOE  in  front  of 
the  eye.  Incident  and  diffracted  rays  even  lie  at  the 
same  side  of  the  normal.  Large  chromatic  and 
monochromatic  aberrations  have  to  be  corrected 
for. 


Fig.  6  :  Optical  concept : 

Integration  of  an  holographic  element  in 
the  visor.  Note  the  large  asymmetry  of 
the  visor  hologram  in  front  of  the  eye. 


Fig.  7  :  Optical  concept : 

Integration  of  an  holographic  element  in 
the  visor.  Projection  to  the  opposite  eye 
reduces  the  HOE  asymmetry. 


A  more  favourable  situation  is  obtained  if  the  rays 
from  the  relay  optics  are  projected  to  the  opposite 
eye  by  means  of  two  HOEs  on  the  visor  (Fig.7). 
This  idea  is  described  in  ref.[4J.  The  asymmetry  at 
the  visor  HOEs  is  highly  reduced,  and  corrections 


of  chromatic  and  monochromatic  aberrations  are 
much  smaller.  Aberrations  originating  from  the 
large  off-  angles  still  have  to  be  corrected  in  the 
relay  optics  and  in  the  recording  configuration  of 
the  HOEs. 

4.4.  Image  intensifier  imaging  sensors  and 
display  aspects. 

The  HMVS  with  its  holographic  visor  is  in 
principle  a  light-weight  and  flight-safe,  multi¬ 
purpose  display  device.  Information  from  different 
imaging  devices  (either  used  separately  or  by 
combination  through  image  fusion)  such  as  Image 
Intensifiers,  FLIR  or  Radar  Imagers  can  be 
combined  with  all  types  of  flight  information  data. 

For  a  fully  integrated  HMVS  only  a  Image 
Intensifier  coupled  directly  to  a  Low  Light  Level 
Television  system,  (LLLTV)  can  be  used  as  an 
on-helmet  imaging  device. 

The  LLLTV,  or  also  called  the  Intensified  CCD 
camera,  gives  the  advantages  of  low-weight, 
minimum  parallax  and  direct  Line  of  Sight  control 
by  movements  of  the  head. 

This  LLLTV  concept  is  described  in  more  detail  in 
ref.  [6j. 

In  this  kind  of  systems,  the  weight,  the  dynamic 
range  and  the  resolution  are  crucial  aspects. 

For  the  LLLTV  system  weight  can  be  minimized 
by  using  plastic  optics,  a  highly  integrated  design 
for  FT  tube  and  High  Voltage  Power  Supply 
(HVPS)  and  finally  by  using  electron-bombarded 
CCD’s.  The  latter  combination  leaves  out  the 
anode  screen  and  tapered  fiber  optics  for  coupling 
the  FT  tube  with  the  CCD  sensor. 

In  this  way  the  additional  weight  for  a  LLLTV 
sensor  can  be  reduced  to  approximately  170 
grams. 

Naturally  the  LLLTV  sensor  can  be  easily 
installed  in  other  places  within  the  airframe  due  to 
the  fact  that  the  image  is  available  as  a  video¬ 
signal. 

As  compared  with  NVG’s,  LLLTV  gives  the  main 
advantages  of  weight  reduction  (less  optics 
required),  optimization  of  center  of  gravity  and 
flight-safety,  and  possibilities  for  image  processing 
and  sensor  fusion. 

A  potential  problem  with  using  LLLTV  sensors 
under  several  conditions  of  illuminance  lies  in 
both  the  overall  dynamic  range  (more  than  a  factor 
of  1000  between  starlight  and  deep  twilight)  and 
the  intrascene  dynamic  range,  which  can  reach 
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Fig.  8  :  Diagram  of  adaptive  LLLTV  System  [ref.  6] 


values  up  to  100.000  between  shadows  and  bright 
spots. 

By  using  a  highly  sensitive  I*T  with  carefully 
selected  non  linear  characteristics  and  adaptive 
control  mechanism,  a  separate  control  of  FT  and 
CCD  gain,  and  an  analysis  of  the  video-output  for 
each  pixel  (see  fig.  8  and  ref.  [6]),  these  problems 
will  be  minimized. 

A  further  upward  extension  in  overall  dynamic 
range  can  be  reached  by  using  an  auto-iris  control 
mechanism  or  by  implementation  of  cathode 
gating. 

Resolution  in  combination  with  the  required  field 
of  view  (at  least  40°  diagonal)  is  not  limited  by 
the  LLLTV  system  but  by  the  video  standards  in 
combination  with  the  use  of  CRT’s. 

A  1"  miniaturized  CRT  (weight  approximately  90 
grams)  is  capable  of  displaying  an  extended  CCIR 
video  signal  (576  vert,  x  768  horizontal  pixels) 
with  sufficient  brightness.  Further  miniaturization 
is  highly  unlikely  due  to  physical  restrictions.  The 
additional  weight  for  a  dual  on-helmet  CRT 
system  (which  has  several  advantages  as  described 
below)  with  a  combined  HVPS  unit  will  add  up  to 
approximately  300  grams. 

High  Definition  Television  (HDTV)  application 
however  will  require  larger  and  thus  heavier 
CRT’s,  which  also  has  its  implications  on  the 
HVPS  for  these  components  and  on  thermal 
dissipation.  Therefore  one  must  very  carefully 
consider  the  actual  operational  requirement  for 
system  resolution. 

The  use  of  dual  CRT  system  is  preferable,  even  if 
only  one  imaging  sensor  is  used,  in  order  to  have 
a  potential  solution  for  both  the  problems  of  dark- 
adaptation  and  binocular  rivalry  (ref.  [7]). 


A  possible  solution  for  the  weight-problem  in  this 
case  lies  in  the  use  of  Fiber  Optics  coupling 
between  the  CRT  and  its  related  electronics  (off- 
helmet)  and  relay  optics  (on-helmet). 


5.  CONCLUSION. 

New  developments  in  holographic  night  vision 
systems  are  described,  with  emphasis  on  the 
holographic  visor  concept.  Integration  of  a  HOE  in 
the  visor  of  the  helmet  improves  the  weight  and 
the  safety  of  the  system,  but  limitations  inherently 
related  with  the  optical  function  of  the  HOEs  and 
the  material  properties  of  the  visor  have  to  be 
taken  into  account  during  tire  design  of  HiviVS. 


A  new  concept  for  helmet  mounted  vision  is 
described.  This  concept  has  a  high  degree  of 
flexibility  by  the  use  of  a  LLLTV  camera. 
Attention  to  the  critical  factors  like  weight, 
dynamic  range  and  resolution  is  given. 
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0  RESUME 

Les  exigences  de  pilotage  et  de  navigation  de  jour 
conune  de  nuit  d’un  a^ronef  de  combat,  contraignent 
l'interface  homme-machine  4  une  Evolution 
permanente.  Les  quinze  demibres  ann6es  ont  vu  les 
collimateurs  "tSte  haute*  se  glnlraliser  sur  les 
avions  de  combat.  Cependant,  malgre  les  techniques 
holographiques  permettant  d’augmenter  la  taille  du 
champ  optique  presente,  la  recherche  portant  sur 
(’amelioration  du  systbme  d’armes  est  passbe  par  la 
creation  d’une  nouvelle  interface  appelde  "  Viseur  de 
casque*  plac£e  devant  1’oeil  du  pilote. 

Cette  interface  permet  au  pilote  de  pointer  les 
capteurs  du  systbme  d’armes  dans  une  direction 
donnde,  et,  inversement  au  Systfeme  de  navigation  et 
d’artnement,  de  montrer  un  objectif.  La  taille  des 
champs  optiques  proposes  pour  les  viseurs  de 
casque,  dquivalente  au  champ  d’un  viseur  tSte  haute 
dassique,  reste  insuffisante  pour  autoriser  le  pilotage 
ou  la  navigation  en  toute  security.  Tout  au  plus 
quelques  parambtres  peuvent-ils  etre  prbsentes  pour 
aider  le  pilote  dans  la  realisation  de  sa  mission. 

Les  techniques  s’ameiiorant,  1'existence  de  ce  type 
d’interface  a  donnb  la  possibility  de  presenter  une 
image  plus  complexe  devant  les  yeux  du  pilote.  II 
s’agit  du  ’Visuel  de  Casque". 

Un  expose  des  exigences  de  pilotage  et  de 
navigation  en  toutes  conditions  mettra  en  evidence 
les  critbres  indispensables  pris  en  compte  pour  la 
conception  de  ces  nouveaux  visuels. 

I  fffTRQPVCTIQN 

Les  avions  de  combat  modemes,  confrontes  k  des 
menaces  accrues  doivent  accomplir  leurs  missions  de 
nuit  comme  de  jour  par  toutes  conditions  meteo. 
Pour  la  reussite  de  ces  missions,  ils  disposent  de 
capacites  d’evolution  accrues,  d’armements  evolues, 
et  de  capteurs  nouveaux  ou  k  capacites  ameiiorees. 

II  convient  done  de  les  rendre  capables  de  tirer 
pleinement  benefice  de  ces  moyens  en  reduisant 
global e men t  les  deiais  d’exploitation  et  en  assurant 
une  couverture  plus  large  de  I’environnement. 
Exploitant  davantage  d’informations  temps  reel,  les 
nouvelles  interfaces  homme-machine  doivent 
permettre  au  pilote  de  reagir  trbs  rapidement,  de 
f>9on  naturelle  et  spontanee,  face  aux  menaces,  et 
ce  queiles  que  soient  les  conditions  jour/nuit 
mauvaise  m6i6o. 


Pour  cela,  il  est  nbcessaire  d’optimiser  1’ integration 
des  systbmes  et  de  dbvelopper  1’ homogeneity  entre 
1’ ensemble  arme men ts -capteurs  et  1’ ensemble  des 
visualisations  sans  degrader  la  vision  directe  du 
monde  exterieur  lorsqu'elie  existe. 

Ce  besoin  a  conduit  k  dbvelopper  des  systbmes  de 
visbe  et  de  visualisation  de  casque  utilisables  quelle 
que  soit  l’orientation  de  la  tSte  du  pilote. 

Cependant,  il  convient  d’analyser  sur  le  plan 
fonctionnel  ce  que  Ton  peut  attendre  d’un  dispositif 
de  visualisation  sur  casque  en  regard  des  exigences 
operationnelles  de  realisation  de  mission. 

2  EXIGENCES  OPERATIONNELLES 

Une  analyse  exhaustive  des  exigences 
operationnelles  conditionne  to  us  les  travaux  entrepris 
sur  les  visuels  de  casque.  Il  est  un  fait  que  le 
pilotage  et  la  navigation  sont  considerablement 
facilitds  lorsque  le  pilote  possbde  une  parfaite 
apprehension  du  monde  exterieur  pour  son 
orientation  spatiale  (horizon  defini,  relief  visible)  et 
son  evolution  au  sein  de  celui-ci  (detection  des 
obstacles  artificiels,  perception  de  1’altitude  et  de  la 
vitesse  relative).  De  plus,  la  realisation  de  missions 
operationnelles  necessite  (’acquisition  visuelle 
precoce  de  points  particuliers  tels  que  les  obstacles 
artificiels  ou  repbres  naturels  pour  ie  recalage  de 
systbme  de  navigation  et  ie  traitement  des  objectifs. 
Aussi,  lors  de  1’execution  des  missions  en  trbs  basse 
altitude  l’image,  presentee  au  pilote  par  le  visuel  de 
casque  devra  recreer  le  plus  fidbiement  possible  les 
conditions  d’un  pilotage  et  de  navigation  k  vue,  avec 
un  enrichissement  symbolique  pour  I’orientation 
spatiale  et  la  security.  L’ information  presentee  ne 
devra  laisser  aucun  doute  quant  k  sa  validity,  rien 
n'etant  plus  discreditant  qu’une  mauvaise 
superposition  de  I' image  reerbee  et  du  paysage, 
lorsqu’il  est  visible. 

L’emploi  de  tels  systbmes  par  1' aviation  de  defense 
abrienne  impose  que  les  missions  d'attaque  air/sol 
s'executent  k  nouveau  en  patrouille.  Tous  les  aspects 
de  tenue  de  place,  d’auto-protection  et  de  security  au 
sein  de  patrouilles  lourdes  se  retrouvent  poses. 

Enfin,  dans  un  tel  contexte,  excessivement  proche 
du  contexte  jour  en  bonnes  conditions 
meteorologiques,  1’equipement  de  tete,  jusqu’alors 
neglige  redevient  une  priority  en  terme  de  masse  et 
de  centrage,  le  facteur  de  charge  augmentant  avec 
les  risques  rencontres. 
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2.1  EXIGENCES  DE  PILOTAGE 
ET  NAVIGATION 

2.1.1  Roulaee 

Cetle  phase  parfois  complete  de  jour  selon  le  terrain 
de  depart  peut  se  reveler  delicate  de  nuit  sur  un 
terrain  sommairement  dclaird.  Les  obstacles  au  sol 
doivent  £tre  localises  et  identifies  k  plus  de  200  m  k 
±  90°  en  avant  de  l’appareil. 

2.1.2  Decollage 

La  trajectoire  de  montee  doit  pouvoir  8tre  ajustee 
pour  profiter  des  obstacles  du  relief  dont  les  grandes 
lignes  doivent  Stre  per$ues  de  3  k  4  km  dormant  un 
preavis  de  !  5  k  20  secondes. 

2.1.3  Trfes  basse  altitude 

Le  besoin  essentiel  est  de  foumir  au  pilote  une 
perception  du  monde  exterieur  aussi  proche  que 
possible  de  celle  qu’il  a  de  jour.  Deux  aspects  sont 
k  considered  La  perception  du  paysage  qui  va  Stre 
survoie  k  court  terme  doit  Stre  aussi  detailiee  et 
lointaine  que  possible  afin  d'optimiser  le  suivi  du 
terrain  et  1’evitement  des  obstacles.  La  perception  du 
paysage  lateral  doit  Stre  assuree  pour  ameiiorer  la 
perception  de  l’environnement  global,  faciliter 
1’orientation  et  tenir  une  place  dans  une  formation. 

Le  vol  k  proximite  du  sol  necessite  une  attention 
soutenue  pour  des  raisons  de  sauvegarde.  Le 
contrdle  des  paramfetres  machine  devient 
particuli&rement  deiicat  et  doit  St  re  execute  sans 
recherche  en  cabine. 

2.1.4  Approchc 

Le  retour  de  mission  est  un  point  deiicat  k  realiser, 
car  si  la  partie  operationnelle  est  executee,  il  n'en 
reste  pas  moins  que  tous  les  avions  en  retour  doivent 
se  presenter  selon  des  procedures  definies,  k  des 
points  geographiques  &  identifier,  en  assurant  une 
anticollision  1  vue  lorsque  les  moyens 
radioeiectriques  ne  sont  pas  actives. 

2.1.5  Atterrissage 

Les  espacements  etant  assures  dans  la  phase 
precedente,  les  conditions  de  visibilite  requises  sont 
du  m£me  type  que  les  conditions  rencontrees  en 
temps  de  paix  soient  classiquement  1000  metres  de 
visibilite. 


2.2  EXIGENCES  DE  VISUALISATION 
DE  MISSION 

2.2.1  Localisation  de  points  remarouables 

Au  cours  de  sa  navigation,  le  pilote  doit  etre  capable 
d’effectuer  des  recalages  de  son  syst&me  de 
navigation  inertielle  sur  des  points  remarquables  par 
visee  unitaire  associde  k  une  mesure  de  distance.  De 
la  m£me  faijon,  en  cours  de  penetration,  le  pilote  est 
4  me  me  de  detecter  des  objectifs  d’opportunite  qu’il 
doit  designer  au  systfcme  pour  en  enregistrer  les 
coordonnees. 

2.2.2  Attaaue  d’obiectifs  sol 

Deux  scenarios  d'attaques  air/sol  (A/S)  sont  & 
distinguer  : 

-  l'appui  aerien  qui  correspond  aux  attaques  A/S 
sur  le  champ  de  bataille  dans  un  environnement 
evoluant  rapidement, 

-  les  missions  d’interdiction  qui  se  ddroulent  en 
arrifere  de  la  ligne  de  front  sur  des  objectifs  fixes 
ou  peu  mobiles  dont  la  position  geographique  est 
connue  avant  le  debut  de  mission. 

2.2.2. 1  Appui-aerien 

Dans  ce  cas,  la  phase  d'attaque  necessite  de  detecter 
1’objectif  dans  le  paysage,  de  le  designer  au  SNA  si 
les  coordonndes  initiates  sont  errondes,  d’utiliser  les 
armements  avec  leurs  performances  maximales,  et 
de  surveiller  le  contexte  environnant  afm  d’assurer 
1’ autoprotection  et  l’anticollision.  L’exdcution  de 
cette  mission  en  toutes  conditions  necessite 
l’existence  de  systfcmes  assurant  les  fonctions  de 
detection  et  d’identification  k  des  distances 
suffisantes  pour  assurer  la  sdcurite  du  chasseur.  La 
perception  du  paysage  environnant  doit  etre 
complete  pour  donner  le  preavis  necessaire  aux 
reactions  face  aux  menaces. 

L’utilisation  des  armements  classiques  dans  ce  type 
de  mission  necessite  de  presenter  en  permanence  au 
pilote  et  en  secteur  frontal  les  informations  relatives 
k  la  conduite  de  tir,  et  de  lui  foumir  une 
representation  stabilisee  du  monde  exterieur. 

2. 2.2.2  Interdiction 

Dans  les  missions  d’interdiction,  les  coordonnees  de 
l’objectif  sont  connues,  la  phase  d'acquisition  est 
assuree  par  le  systfeme  d’armes  qui  indique  au  pilote 
la  position  de  celui-ci. 
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L’utilisation  de  tous  types  d’armement  ndcessite 
avant  tout  un  excellent  recalage,  l'objectif  n’dtant 
pas  survold  lors  de  l’emploi  d  armements  moderaes. 

Enfrn  l’autoprotection  au  sein  de  la  patrouiile 
ndcessite  la  perception  du  monde  environnant,  afin 
de  donner  un  preavis  suffisant  pour  engager  une 
manoeuvre  de  defense. 

2.3  QUANTIFICATION  DES  BESOINS 

Hormis  les  besoins  chiffrds  lors  des  phases  de 
roulage  et  d’atterrissage,  les  chiffres  commundment 
affichds  lors  des  phases  de  navigation  et  d'attaques 
sont  des  consequences  de  la  vitesse  de  penetration. 
Une  visibility  .fordsentant  15  k  20  secondes  de  vol 
donne  un  minimuni  cn  duquel  la  mission  ne 
peut  etre  effectuee  It  vue. 

Les  besoins  specifiques  k  la  mission  operationnelle 
dependent  du  preavis  necessaire  it  une  reaction 
coordonnee  et  plannifiable : 

-  la  detection  d’obstacles  de  type  pylones  de  lignes 
"haute-tension"  et  d’antennes  (50  x  3  m)  de  4 

St  5  km, 

-  la  detection  et  la  reconnaissance  de  menace 
Sol/ Air  ou  Air/Air  (3  x  3  m)  k  une  distance 
suffisante  pour  minimiser  les  risques  soil  de 
l'ordre  de  4  &  5  km, 

-  la  detection  et  la  reconnaissance  des  points  de 
recalage  (5  x  5  m)  1  une  distance  compatible  de 
1’intervisibiiite  Air/Sol  et  des  possibilites  de 
mesure  de  distance  typiquement  3  km  avec 
recalage  entre  500  et  1000  m  en  secteur  avant, 

-  la  detection  et  la  reconnaissance  des  cibles  a  traiter 
(dim.  typ.  10  x  10  m)  4  une  distance  compatible 
de  I’engagement  d'une  conduite  de  tir  soit 

*  10  km 

-  la  precision  &  obtenir  pour  le  pointage  d’un  capteur 
doit  permettre  d’atteindre  une  cible  de  5  m  4 
1000  m. 

-  le  debattement  angulaire  de  la  ligne  de  visee  doit 
etre  compatible  de  i’autoprotection  done  atteindre 
les  valeurs  de  ±  150°  en  gisement  et  +  90°/-  60° 
en  site. 


2.4  EXIGENCES  ERGONOMIOUES  (Ref.  13 

Donner  aux  pilotes  la  possibilite  d’effectuer  de  nuit 
des  missions  jusqu’ici  realisees  de  jour,  implique  la 
presence  d’une  defense  adrienne  plus  dense.  La 
presence  d’un  visuel  de  casque  ne  doit,  dans  ce 
cadre,  en  aucun  cas  entrainer  de  g£ne  ergonomique. 

Sans  reprendre  en  detail  toutes  les  conditions  h 
imposer,  l’equipement  de  tSte  ne  devrait  pas  exedder 
la  masse  des  dquipements  de  tete  actuellement  en 
service  dans  les  forces  adriennes.  II  sera 
parfaitement  dquilibrd  pour  permettre  au  pilote  de 
subir  les  accelerations  d’evolutions  (-1  <  G  <  5) 
en  basse  altitude  et  les  accelerations  d’ejection. 

Le  champ  visuel  accessible  doit  etre  au  plus  proche 
du  champ  visuel  normal,  les  masques  visuels  doivent 
etre  limites  au  maximum.  La  presence  d’eiements 
optiques  devant  l’oeil  est  psychologiquement  mal 
acceptde  par  les  pilotes  pour  des  raisons  de  security 
en  cas  d’exposition  accidentelle  au  vent  relatif. 

Tout  en  dtant  confortable,  l’equipement  de  tete  doit 
assurer  toutes  les  functions  de  protection 
physiologique  normale  (oxygbne,  surpression 
obligatoire)  ainsi  qu’une  protection  au  risque  NBC, 
avec  le  minimum  de  gene. 

2.5  EXIGENCES  DTNTEGRATION 

Ces  exigences  couvrent  1’aspect  emploi  d’un  visuel 
de  casque  dans  une  cabine  dvolude  en  terme  de 
compatibilite  avec  les  capteurs  et  visualisations 
existants.  Tous  les  capteurs  doivent  etre  associds 
pour  foumir  une  information  opdrationnelle 
complete,  enrichie  par  le  systbme  d’armes. 

La  vision  directe  du  pilote  ne  doit  pas  etre  altdrde 
par  la  presentation  d’images  issues  d’intensificateurs 
de  lumidre  ou  de  senseur  infrarouge,  que  ce  soit  du 
paysage  extdrieur  (majoritd  du  temps)  ou  de  la 
planche  de  bord  pour  la  recherche  d’informations 
particulidres. 

La  superposition  avec  les  informations  prdsentdes 
dans  les  Collimateurs  tete  haute  doit  etre 
particulidrement  dtudide. 

3  SOURCES  D’lMAGEREE  DISPONIBLES 

Des  exigences  opdrationnelles,  nous  pouvons  k 
l’dvidence  dnoncer,  les  besoins  en  images  pour 
presenter  au  pilote  un  paysage  aussi  proche  que 
possible  de  la  rdalite.  Plusieurs  techniques  sont 
disponibles  pour  donner  une  representation  du 
monde  extdrieur  par  toutes  conditions. 
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L'intensification  de  lumifere  et  l’imagerie  infra-rouge 
represented  les  deux  techniques  les  plus  courantes. 

Elies  peuvent  St  re  complStSes  par  les  techniques 
d’imagerie  radar  millimStrique  et  de  numSrisation  de 
terrain.  Ces  deux  demiSres  sources  d’images  ne 
seront  pas  aborddes  dans  cet  article,  mais  il  faut 
ndanmoins  admettre  qu’une  representation  fiddle  du 
monde  extdrieur  passera  par  1’utilisation  et  la 
synthdse  de  toutes  les  images  issues  de  capteurs 
compldmentaires. 

Enfin,  l’enrichissement  de  1’information  est  rdalisd 
par  t’emploi  d’une  symbologie  appropride. 

3.1  INTENSIFICATION  DE  LUMIERE  :  IL 

Le  principe  utilise  consiste  &  recueillir  puis  amplifier 
la  trds  faible  lumidre  rdsiduelle  toujours  prdsente, 
mSme  la  nuit,  et  &  presenter  &  l’oeil  de  l’observateur 
une  image  interpretable.  Le  fonctionnement  des 
yquipements  IL  repose  sur  la  propriety  des 
materiaux  constitutifs  du  ddtecteur  (photocathode)  de 
convertir  en  electrons  les  photons  provenant  de  la 
scdne  et  collectdes  par  I’objectif  d'entrde  ;  ceux-ci 
sont  accdierds  et  multiplies  dans  un  tube  &  vide  par 
un  champ  dlectrique  puis  excitent  un  Scran 
luminescent  pour  dormer  une  image  visuelle  (ref.  2). 

Trois  generations  de  tubes  intensificateurs  de 
lumidre  reposant  sur  des  composants  diffdrents  ont 
successivement  dtd  ddveloppdes.  La  plus  rdcente, 
dite  de  3dme  generation,  utilise  un  materiau  pour  la 
photocathode  qui  permet  d’amdliorer  le  rendement 
de  conversions  photons/electrons  d’un  facteur  3.  Le 
deplacement  du  spectre  de  sensibility  permet 
egatement  de  bdneficier  des  caractdristiques 
spectrales  du  rayonnement  nocturne  qui  est  plus 
important  dans  le  proche  infra-rouge  et  du 
comportement  des  scdnes  (en  particulier  la 
vegetation)  dont  le  coefficient  de  reflexion  croit  avec 
la  longueur  d’onde. 

Les  performances  des  tubes  IL  permettent  d’obtenir 
une  image  exploitable  pour  des  conditions 
d’Sclairement  supdrieures  k  1  mix,  ce  qui 
statistiquement  reprdsente  environ  50%  des 
conditions  de  nuit. 

Les  dispositifs  de  vision  nocturne  il  IL  presented  un 
faible  encombrement,  une  simplicite  d'utilisation,  la 
possibilite  d’etre  installs  sur  le  casque  et  enfin  ils 
produisent  une  image  naturelle  proche  de  la  vision 
humaine. 


Ils  presented  cependant  quelques  limitations  ddes  k 
leur  sensibility  aux  conditions  de  luminosity  et  de 
visibility,  les  portyes  et  rysolutions  res  tan  t  limityes. 

L'intensification  de  lumifere  est  utilisde  soit 
directement  (Jumelles  de  Vision  Nocturne,  ou 
J.V.N),  soit  sous  forme  d’un  signal  video  lorsque  le 
tube  IL  est  associy  k  un  dispositif  CCD  (ILCCD). 
Dans  les  applications  visuels  de  casque,  la  solution 
habituellement  retenue  est  le  couplage  optique  entre 
l’intensificateur  de  lumidre  et  l’oeil,  essentiellement 
pour  des  raisons  de  security,  P image  parvenant 
directement  au  pilote  sans  necessiter  de  traitement 
yiectronique.  Cependant,  les  dyveloppements  en 
cours  dans  le  domaine  des  ILCCD  laissent  prdsager 
1’utilisation  de  tels  dispositifs  dans  un  futur  proche 
(ref.  3). 

Les  ILCCD  ont  pour  avantage  la  simplification  du 
schema  optique,  une  reduction  de  la  masse  de 
l'yquipement  et  la  possibility  d’effectuer  des 
traitements  d’image,  en  particulier  la  fusion  avec 
d’autres  sources. 

3.2  IMAGERIE  INFRAROUGE  (IRi 

Le  principe  utilise  consiste  k  capter  et  amplifier  le 
flux  emis  par  tout  corps  dont  la  temperature  est 
diffirente  du  zero  absolu,  et  de  restituer  une  image 
de  type  video.  Compte-tenu  des  caracteristiques 
spectrales  de  1’atmosphdre  et  de  la  temperature  des 
scbnes  k  observer,  les  bandes  spectrales  utilisees 
sont  l’infrarouge  moyen  (3  k  5  /im)  ou  lointain  (8  i 
12  fim).  Les  capteurs  infrarouges  actuellement 
disponibles  sont  constitues  d’une  mosaique 
d’eiements  sensibles,  yventueliement  associees  &  un 
dispositif  de  balayage  optomdcanique  et  ndcessitent 
un  refroidissement  h  77 °K.  Il  en  resulte  que  les 
capteurs  IR  sont  des  dipositifs  encombrants  et 
coflteux. 

Ces  yquipements  d’imagerie  infrarouge  ont  de  trfes 
bonnes  performances  de  sensibility  et  de  rysolution 
permettant  d’obtenir  des  portyes  yievyes.  De  plus,  le 
rayonnement  IR  est  peu  sensible  aux  brumes, 
fumyes  et  poussidres  en  suspension  Hans 
{’atmosphere,  ce  qui  rend  possible  l’utilisation  de 
ces  systfemes  non  seulement  de  nuit  mais  ygalement 
de  jour  par  visibility  ryduite. 

Ces  yquipements  peuvent  se  monter  sur  une 
plateforme  et  ainsi  St  re  asservis  aux  mouvements  de 
la  tete  du  pilote. 
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Les  principals  limitations  sont  leur  encombrement 
et  leur  coflt  qui  rdduisent  le  champ  ({’application. 
Les  capteurs  1R  sont  par  ailleurs  sensibles  k  la 
quantity  d’humiditd  prdsente  dans  I'atmosphkre. 

3.3  COM PLEM ENT ARITE  IR/IL 

Les  conditions  difficiles  pour  les  dispositifs  k  IL 
sont  les  nuits  sans  lune  (obscurity  profonde)  ou  les 
visibility  rdduites.  De  part  ces  conditions,  la 
disponibilitd  opdrationnelle  d’un  tel  systfeme  est 
limitee  k  50%. 

Dans  le  cas  des  capteurs  IR,  les  conditions  difficiles 
sont  les  atmospheres  chaudes  et  humides,  ou  les 
scenes  prdsentant  un  faible  contraste  thermique 
(scenes  "lavees"  par  la  pluie  par  exemple).  La 
disponibilitd  opdrationnelle  de  ces  dquipements,  en 
climat  Centre-Europe,  peut  etre  estimde  k  60%. 

Compte  tenu  de  la  non-correlation  entre  les  causes 
de  mise  en  ddfaut  de  ces  deux  types  d'dquipement  et 
de  la  relative  inddpendance  des  facteurs  de 
comportement,  il  est  gdndralement  ad  mis  que  le  taux 
de  disponibilitd  k  pleines  performances  d’un  systeme 
associant  ces  deux  techniques  est  de  plus  de  80% 
(ref.  4). 

3.4  GESTION  DES  SOURCES  D’lMAGERIE 

Pour  presenter  au  pilote  une  image  du  paysage 
environnant  en  vraie  direction,  les  deux  sources 
d’imagerie  envisages  doivent  etre  asservies  aux 
mouvements  de  la  tete  du  pilote. 

La  fonction  intensification  de  lumikre  met  en  oeuvre 
des  elements  de  volume  et  de  masse  rdduits  pouvant 
St  re  intdgrds  dans  1’dquipement  de  tete.  Ce  principe 
permet  de  rdaliser  simplement  l’asservissement  en 
direction  et  permet  d'autre  part,  en  integrant  deux 
tubes  IL,  de  presenter  une  image  spdcifique  sur 
chaque  oeil,  restituant  la  perception  binoculaire 
naturelle. 

L’imagerie  infrarouge  met  en  oeuvre  un  capteur 
volumineux  qui  ne  peut  Sire  intdgrd  qu’au  niveau  de 
I’avion,  soit  en  interne  dans  la  cellule,  soit  en 
emport.  La  presentation  en  vraie  direction  dans  un 
visuel  de  casque  d’une  imagerie  IR  ndcessite  done 
de  disposer  sur  le  porteur  d’un  systdme  k  ligne  de 
visde  mobile  couplde  k  un  dispositif  de  detection  de 
position  et  d’orientation  restituant  en  temps  rdel  la 
direction  visde  par  le  pilote.  Un  capteur  IL  CCD 
monte  sur  une  plateforme  ndeessiterait  dgalement 
1’emploi  du  mdme  dispositif. 


Enfin,  la  compensation  en  roulis  des  images  peut 
etre  rdalisde  mais  reste  un  sujet  controversd. 
L’intdret  de  la  compensation  des  images  en  fonction 
du  roulis  de  la  tSte  du  pilote  est  de  permettre  une 
superposition  image/paysage.  Il  faut  toutefois  noter 
que  les  conditions  d’ambiance  lumineuse  ndeessitant 
une  bonne  superposition  sont  rares. 

En  l’absence  de  conclusions  formelles  sur  ce  sujet, 
il  semble  ndeessaire  de  valider  1’intdret  de  cette 
compensation  au  cours  d’une  evaluation  en  vol. 

4  REPONSE  TECHNIQUE 

Le  but  de  ce  chapitre  est  de  ddcrire  la  fa^on  dont  le 
concepteur  du  viseur  de  casque  traduit  le  besoin 
opdrationnel,  exprimd  par  l’utilisateur,  en 
specifications  techniques.  On  s’est  attache  k  mettre 
en  evidence  les  paramdtres  les  plus  importants  et  k 
montrer  comment  ils  influent  sur  l’ensemble  du 
projet. 

4.1  FONCTIONS  NECESSAIRES 

Les  fonctions  ndeessaires  pour  remplir  les  besoins 
opdrationnels  ddcrits  prdeddemment  sont : 

-  visualisation  d'une  image  du  paysage  extdrieur 
en  vraie  direction  provenant  d’une  source 

d’ image  appropride  (IR  ou  IL) 

-  visualisation  d’une  image  symbolique  prdsentant 
les  principaux  paramdtres  du  SNA 

-  mesure  de  la  ligne  de  visde  du  pilote 

-  protection  physiologique  du  pilote. 

La  fonction  de  visualisation  permet  de  presenter  au 
pilote  une  image  conforme  au  paysage  observe. 

La  visualisation  d’une  image  synthdtique  satisfait  le 
besoin  de  presentation  au  pilote  des  informations 
ndeessaires  k  la  conduite  de  sa  machine,  k  la  mise 
en  oeuvre  d’une  conduite  de  tir  et  k  la 
comprehension  de  la  situation  tactique. 

La  fonction  de  mesure  de  la  ligne  de  visde  du  pilote 
permet  d’une  part,  de  designer  un  objectif  au  SNA 
et  d’autre  part  d'asservir  une  plateforme  d’imagerie 
situde  en  emport. 

La  protection  physiologique  du  pilote  lui  permet 
d ’assurer  sa  mission  en  toute  sdcuritd. 


13-6 


4.2  CONTRAINTES  GENERALES 

Les  exigences  opdrationnelies  permettent  de  ddfinir 
des  para  metres  incontournables  : 

-  les  besoins  en  portdes  se  traduisent  en  terme 
de  resolution  par  des  valeurs  d’environ  1  mrd 
pour  la  detection  et  0,3  mrd  pour  la 
reconnaissance. 

-  le  champ  visuel  offert  doit  couvrir  deux 
besoins  :  d’une  part  une  resolution  maximale 
dans  un  domaine  angulaire  supdrieur  au  champ 
de  vision  foveale  de  l’oeil  soil  environ  5°, 
d’autre  part,  une  resolution  moindre  dans  un 
champ  le  plus  proche  possible  du  champ  visuel 
naturel  afin  de  percevoir  les  grandes  lignes  du 
paysage  et  les  mouvements  latdraux. 

Cette  multiresolution  n’est  pas  encore  accessible, 
aussi,  la  valeur  commundment  admise  du  champ  i 
presenter  est  comprise  entre  30°  et  40°.  Cette  plage 
represente  le  meilleur  compromis  entre  le  critfere 
’facteur  humain",  la  resolution  necessaire  et  la 
masse  admissible  de  l’dquipement  de  tSte. 

La  meilleure  information  est  obtenue  par  l’utilisation 
de  la  binocularite  (ref.  3). 

Cette  solution  presente  un  certain  nombre 
d’avantages.  Elle  minimise  les  risques  de  rivalite 
oculaire  susceptibles  d’intervenir  lorsque  les  deux 
yeux  voient  des  images  sensiblement  differentes,  en 
particulier  en  cas  de  presentation  d’une  image 
complete  devant  un  oeil  et  de  l’obscurite  devant 
1’autre.  Les  consequences  de  cette  rivalite  peuvent 
Sire  une  perte  complete  momentanee  ou  prolongee 
de  l'image,  pouvant  conduire  k  une  ddsorientation  de 
1’utilisateur.  La  presentation  de  l’image  nocturne 
simultanement  devant  les  deux  yeux  est  done  un 
gage  de  securite  imperatif  pour  un  pilote  d’avion 
d’arme. 

Cependant,  celle-ci  est  soumise  k  1’emploi  de  deux 
capteurs  identiques  et  independants.  Si  elle  est  de 
fait  par  1’emploi  de  2  IL,  elle  n’est  pas  envisageable 
en  terme  d'encombrement  et  de  coflt  pour  les 
senseurs  infrarouges.  Dans  ce  cas,  la  biocularite 
sera  prefer. 

Les  performances  de  la  vision  bio/binoculaire  sont 
supdrieures  k  celles  de  la  vision  monoculaire,  en 
termes  de  sensibilite  au  contraste  (gain  40  *),  de 
perception  de  la  luminosity  et  de  perception  du  bruit 
lorsque  les  deux  capteurs  prdsentent  des  bruits 
spatialement  ddcorrdlds. 


4.3  PRINCIPE  DE  PRESENTATION  D’lMAGES 

L’image  du  paysage  extdrieur  est  prdsente  sur  les 
deux  yeux  simultandment  en  binoculaire  pour  la 
source  IL,  en  bioculaire  pour  la  source  IR. 

La  presentation  en  bio/binoculaire  d’une  seine 
presente  des  contraintes  de  positionnements  absolu  et 
relatif  des  images  devant  les  yeux. 

4.3.1  Tolerances 

II  est  essentiel  que  le  champ  de  l'image  presentee 
soit  situe  autour  de  la  position  centrale  du  champ 
visuel  naturel.  Les  tolerances  de  positionnement 
relatif  des  deux  images  doivent  St  re  respectees  afin 
de  ne  pas  creer  des  problimes  de  fusion  d’images  ou 
plus  simplement  de  gene  apris  un  temps  plus  ou 
mo  ins  long  de  port  de  1’dquipement.  Une  etude 
approfondie  effectude  par  le  "Armstrong  Aerospace 
Medical  Research  Laboratory’  (ref.  6)  a  permis  de 
quantifier  ces  tolerances.  Les  conclusions  sont 
rdsumdes  ci-dessous. 

-  ddsalignement  horizontal 

<  2,3  mrd  en  convergence 

<  1  mrd  en  divergence 
•  ddsalignement  vertical 

<  1  mrd 

-  difference  de  grandissement 

<  3  mrd  sur  l’axe  vertical 

-  difference  en  rotation 

<  1  mrd  sur  1’axe  vertical 

-  difference  de  luminance 

<  25* 

Ces  donnees  s’appliquent  essentiellement  k  la 
symbologie.  Les  valeurs  k  prendre  en  compte  pour 
la  presentation  d’images  sont  moins  contraignantes. 

Si  la  demifcre  donnee  est  relativement  facile  h 
respecter  aussi  bien  avec  une  image  viddo  projetde 
k  partir  des  deux  tubes  &  rayons  catbodiques  qu’avec 
des  IL,  il  n’en  va  pas  de  mdme  des  contraintes 
d’alignement.  En  effet,  ces  contraintes  doivent  fttre 
respeetdes  dans  toute  la  plage  de  rdglage 
interpupillaire  des  optiques.  Elies  impliquent  la  mise 
en  oeuvre  de  mecanismes  robustes,  sans  jeu 
mdcanique  avec  un  deplacement  identique  vers  la 
gauche  ou  la  droite.  Bien  entendu,  les 
caractdristiques  doivent  dtre  atteintes  en  restant  dans 
une  enveloppe  de  masse  minimale,  alors  que  les 
contraintes  impliquent  plus  qu’un  doublement  de 
masse  entre  une  version  monoculaire  et  une  version 
binoculaire. 


13-7 


4.3.2  Champ 

La  valeur  du  champ  rdel  influe  sur  de  nombreux 
aspects  ergonomiques,  et  en  premier  lieu  sur  la 
masse.  En  premiere  approche,  on  peut  dire  que, 
toutes  choses  dtant  dgales  par  ailleurs,  10°  de  champ 
visual  correspondent  k  un  supplement  de  masse  de 
200g.  La  realisation  des  fonctions  opdrationnelles 
peut  dtre  accomplie  avec  un  champ  de  vision  limite 
k  30°.  Une  valeur  de  champ  supdrieure  permettrait 
d’amdliorer  le  confort  visuel  du  pilote  en  dlargissant 
sa  vision  peripherique.  Cette  augmentation  de  champ 
ne  pourrait  etre  envisagde  qu’k  condition  que  les 
performances  de  resolution  des  capteurs  d’imagerie 
aug  men  tent  dans  Ie  me  me  rapport. 

Cependant,  l’adjonction  d’une  visualisation  situde  en 
pdripberie  du  champ  visuel  apportera  un  complement 
d’informations  pour  1'orientation  spatiale. 

4.3.3  Pupille  de  sortie 

La  pupille  de  sortie  est  ddfinie  comme  dtant  la  zone 
de  ddplacement  de  1’oeil  par  rapport  au  dispositif 
optique  dans  laquelle  la  vision  de  rintdgralite  de 
l’image  est  conservde.  Cette  notion  est  aussi 
importante  que  celle  de  champ. 

Pour  la  valeur  de  champ  spdcifide,  la  dimension  de 
pupille  minimale  &  retenir  est  10  mm  de  faqon  k 
couvrir  le  ddbattement  de  1’oeil  correspondant  It  une 
vision  en  bord  du  champ  de  l’image. 

En  considdrant  les  mouvements  rdsiduels  du  casque 
sur  la  tdte,  le  diambtre  de  la  pupille  de  sortie  du 
dispositif  optique  doit  done  au  moins  etre  dgal  k 
13  mm. 

Le  diambtre  de  pupille  doit  dtre  respeetd  non 
seulement  au  centre  du  champ  mais  encore  en  bord 
de  champ.  Tout  comme  le  champ,  une  augmentation 
de  la  pupille  a  un  impact  direct  sur  la  masse 
(ref.  7). 

4.3.4  Tirage  d’anneau 

Le  tirage  d’anneau  est  ddfini  comme  dtant  la 
distance  minimale  entre  l’oeil  et  l’dldment  optique  le 
plus  proche.  Une  valeur  minimale  de  25  k  30  mm 
est  suffisante  pour  assurer  la  compatibilitd  avec  le 
port  de  lunettes  de  correction  ou  avec  un  dquipement 
de  protection  NBC. 

4.3.5  Transmission  de  la  voie  directe 

Une  transmission  importante  de  la  voie  directe  a  un 
impact  immddiat  sur  la  luminance  de  sortie  des 
sources  d’images.  Une  valeur  de  30%  est  suffisante 
pour  le  pilotage  de  nuit. 


4.4  IMAGERIE 

4.4.1  Format 

Le  besoin  k  satisfaire  est  d’  assurer  la  compatibilitd 
du  viseur  de  casque  vis  k  vis  du  standard  viddo  des 
capteurs  d’imagerie  disponibles. 

Le  viseur  de  casque  devra  done  dtre  compatible  de 
la  prdsentation  d’une  image  viddo  conforme  au 
STANAG  3350  B  (625  lignes,  50  Hz,  format  4/3). 

4.4.2  Luminositd 

La  luminositd  de  1’ image  viddo  prdsentde  doit  dtre 
compatible  de  1’exploitation  par  le  pilote  qu’elles 
que  soient  les  conditions  d’ ambiance  lumineuse. 

Elle  doit  d’autre  part  dtre  rdglable  doit  dtre  rdglable 
continuement.  Un  asservissement  automatique  de 
luminositd  est  souhaitable. 

4.4.3  Contraste 

L’image  viddo  prdsentde  dans  le  casque  doit  disposer 
d’un  contraste  suffisant  k  l’interprdtation. 

Le  contraste  de  I’ image  viddo  doit  dtre  supdrieur  k 
5,6:1  pour  garantir  la  visualisation  d’au  moins  6 
niveaux  de  gris. 

4.4.4  Rdsolution 

Les  besoins  de  rdsolution  ndeessaires  pour  la 
ddtection  et  la  reconnaissance  sont : 

-  ddtection  :  rdsolution  >  1  cycle/mrd 

-  reconnaissance  :  rdsolution  >  3  cycles/mrd 

Les  deux  capteurs  utilisds  (IL-IR)  couvrent  le 
besoin.  La  ddtection  est  assurde  par  les  IL  et  les 
capteurs  1R,  la  reconnaissance  par  les  capteurs  IR. 
Dans  ces  conditions,  les  images  utiles  pour  la 
reconnaissance  seront  prdsentdes  sur  une 
visualisation  annexe  oil  la  fonction  agrandissement 
sera  utilisable. 

Pour  des  raisons  de  sdcuritd,  seule  une  image  k 
l’dchelle  1  sera  prdsentde  en  visuel  de  casque.  Les 
risques  de  ddsorientation  spatiale  sont  importants 
lors  de  la  ddcorrdlation  de  l’dchelle  de  l’image  et  de 
la  rdalitd,  mdme  si  la  sdlection  de  l’image  est 
commandde  par  un  sdlecteur  instable. 
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4.4.5  Svmbologie 

Contraste  et  resolution  caracterisent  la  symbologie. 

Le  contraste  est  defini  par  le  rapport  suivant  : 

C  =  (LI  +  L2)/L1 

avec  LI  la  luminosity  du  fond  (image  video  ou 
paysage  exterieur)  et  L2  la  iuminosite  des  symboles. 
Les  performances  de  resolution  de  l’oeil  necessitent 
de  disposer  d'un  contraste  des  symboles  par  rapport 
au  fond  supdrieur  k  1,3:1. 

Conceraant  la  resolution,  la  largeur  de  trace  des 
symboles  sera  de  1,0  ±  0,2  mrd. 

Enfin,  la  position  du  reticule  de  visde  sera  ajustable 
dans  le  champ  afin  de  determiner  experimentalement 
la  position  optimale  pour  une  application  Air/Sol  sur 
avion  d’armes. 

4.5  PERFORMANCES  DE  MESURE  DE  LA 
L10NE  DE  V1SEE 

4.5.1  Debattements  aneulaires 

Les  besoms  operationnels  recensds  pour  les  capacitds 
de  debattement  angulaire  de  la  tete  du  pilote  sont  les 
suivants  : 

-  Visdes  Air/Sol  : 


Gisement 

±  90° 

Site 

-  60°,  + 

Roulis 

±  30° 

Autoprotection  : 

Gisement 

±  150° 

Site 

-  60°,  + 

Roulis 

±  45° 

4.5.2  Precision 

Les  besoins  en  precision  statique  sur  la  mesure  de  la 
ligne  de  visee  du  pilote  sont  de  deux  types  : 

-  designation  d’un  objectif  :  0,25° 

-  asservissement  d’un  capteur  IR  :  la  precision 
ndcessaire  doit  St  re  determinee  en  fonction  du 
capteur  mis  en  oeuvre  et  des  caracteristiques  de  la 
chaine  d ’asservissement. 

La  precision  k  obtenir  dans  la  superposition  des 
images  avec  le  monde  reel  reste  it  definir 
experimentalement.  Les  ecarts  admissibles  sont  ceux 
qui  n’engagent  pas  la  security  et  ne  discredited  pas 
1’information  lore  de  la  vue  simultanee  des  images 
capteure  et  du  monde  reel. 


De  la  meme  fa<;on,  la  precision  dynamique  it  obtenir 
depend  des  capteure  utilises  pour  les  fonctions  de 
designation  (radar  ou  autodirecteur)  et  du  capteur  IR 
asservi.  Une  precision  dynamique  de  1,3°  it  95% 
pour  un  mouvement  k  30°/s  peut  Sire  retenue  it 
priori. 

4.6  CASQUE  SUPPORT  DU  VISUEL 

La  prise  en  compte  simultande  de  toutes  les 
exigences  opdrationnelles  de  protection 
physiologique  et  d’ergonomie  a  permis  de  mettre  en 
evidence  l’existence  d’une  solution  originate,  le 
casque  integral. 

4.6.1  Protection  physiologique 

La  protection  aux  chocs  et  la  resistance  it  la 
perforation  sont  excessivement  bien  assurees.  11  en 
va  de  meme  pour  la  protection  faciale  et  oculaire  en 
cas  d’ exposition  au  souffle  aerodynamique  jusqu’k 
625  Kts. 

La  protection  respiratoire  comprend  d’une  part 
1'oxygenation  du  pilote  et  d’autre  part  les  fonctions 
de  surpression  respiratoire  en  fonction  du  facteur  de 
charge  ou  en  cas  de  depressurisation  cabine. 

La  protection  visuelle  est  assur6e  contre  le  soleil 
lorsque  ndcessaire,  les  reflets  ou  les  rayonnements 
laser. 

L'isolation  phonique  procurde  est  identique  it  celle 
des  casques  standards.  Cependant,  (’utilisation  de 
filtres  actifs  peut  6tre  envisagde. 

La  formule  "Casque  integral”  facilite 
considerablement  la  protection  NBC  tout  en 
ma intenant  toutes  les  capacitds  operationnelles. 

4.6.2  Ergonomie 

La  masse  de  l’equipement  de  tete  est  mini  mi  see  de 
fa;on  k  eviter  toute  fatigue  ou  gene  pour  le  pilote. 
Selon  le  masque  utilise,  la  masse  est  comprise  entre 
1,8  et  2  kg. 

La  position  du  centre  de  gravity  resultant  (tete  + 
equipement  de  tete)  est  conforme  aux  tolerances 
admises  k  l’heure  actuelle. 

Le  casque  poss&de  d’autre  part  une  pe normalisation 
adaptee  k  chaque  pilote.  EUe  participe  k  la  tenue  de 
1’  ensemble  tete-casque  tout  en  assurant  un  excellent 
confort. 

Le  principe  de  visualisation  retenu  etant  de  type 
binoculaire,  un  reglage  du  module  optique  pennet 
d’adapter  1’equipement  de  tete  k  l’6cart 
interpupillaire  du  pilote. 


Lr<  Visuel  de  Casque  ainsi  dcfir.i  rfpond  aux 
exigences  opErationnelles.  II  donne  au  pilote  les 
moyens  de  contrdler  son  appareil  et  rEaliser  sa 
mission  par  tout  temps  dans  de  bonnes  conditions  de 
sEcuritE. 

Cependant,  et  pour  ces  memes  raisons,  la 
reconnaissance  des  objectifs  nEcessitant  des 
changements  d’Echelle  des  images  ne  pourra  Etre 
assurEe  que  dans  une  visualisation  moyenne  ou 
latErale. 

Cette  solution  reprEsente  le  compromis  idEal 
aujourd’hui  en  regard  de  l’Etat  de  l’art  en  la  matikre. 

Les  Evolutions  futures  du  visuel  de  casque  devront 
prendre  en  compte  les  caractEristiques 
pbysiologiques  du  capteur  rEtinien  pour  asservir  la 
rEsolution  de  1’image  prEsentEe  k  la  position  de 
l’oeil.  Cette  solution  permettra  de  toujours  presenter 
au  pilote  1’ image  de  plus  haute  rEsolution  sans  pour 
autant  nEcessiter  des  capacitEs  et  des  vitesses  de 
calcul  accrues. 

Les  difFErents  capteurs  ou  bases  de  donnEes 
embarquEs  devront  participer  k  la  crEation  de 
1’ image  pour  foumir  au  pilote  un  seul  type  d’images 
fusionnEes  plus  ou  mo  ins  enrichies  en  fonction  des 
capacitEs  instantanEes  de  chaque  senseur. 

Mais  k  ce  stade  se  posera  la  question  de  la 
superposition  prEcise  des  images  issues  des  capteurs 
et  du  monde  rEel.  Une  mesure  beaucoup  plus  prEcise 
de  la  ligne  de  visEe  peut,  en  valeur  absolue, 
apporter  la  solution.  Cependant  les  frEquences  de 
calcul  de  cette  ligne  de  visEe  ne  vont  elles  pas  crEer 
un  obstacle  dEterminant  ? 

Dans  quelle  mesure  cette  superposition  parfaite  est- 
elle  nEcessaire,  surtout  si  les  conditions  ne 
permettent  pas  la  vue  de  paysage  ? 

DEs  lors  que  le  champ  utilisE  par  les  dififErents 
dispositifs  optiques  permettra  de  foumir  au  pilote  les 
alettes  en  champ  visuel  pEriphErique,  et  que  les 
capteurs  couvriront  le  mEme  espace  que  1’oeil  du 
pilote  avec  une  prEcision  de  la  ligne  de  visEe  ne 
mettant  pas  en  jeu  la  sEcuritE,  alore  il  sera  possible 
d’envisager  l’utilisation  du  cockpit  fermE. 


Cependant,  dans  ce  contexte,  le  pilote  devra  tout 
de  mEme,  Eire  capable  de  se  construire  au  mieux 
sa  propre  representation  mentale  de  la  situation 
sans  laisser  au  systkme,  le  so  in  de  dEcider  pour 
lui. 

Quelles  images  doit-on  prEsenter  au  pilote  ?  Quelle 
redondance  de  moyens  faut-il  lui  foumir  pour  lui 
assurer  le  jugement  d’une  situation  donnEe  et  les 
dEcisions  en  dEcoulant  ? 

Ces  grandes  questions  posEes  n’empechent 
cependant  pas  de  penser  que  le  cockpit  fermE  ou 
virtuel  n’est  plus  aujourd’hui  inacessible. 
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1.  SUMMARY 

As  a  result  of  progressively  adding  on 
capabilities  to  the  basic  protective 
flying  helmet,  present  solutions  to 
operational  requirements  result  in 
barely  acceptable  flying  helmet 
assemblies.  The  devices  which  provide 
these  capabilities  must  all  be 
compatible  with  each  other,  with  the 
rest  of  the  pilot's  Aircrew  Equipment 
Assembly  ( AEA)  and  with  the  aircraft. 
Often  conflicting  requirements  result  in 
compromises  being  made.  Fundamental 
rules  of  optics  cannot  be  changed,  and 
in  general,  improved  optical  performance 
results  in  larger,  heavier  head-borne 
load. 

Heretofore,  the  solutions  have  been 
engineered  by  several  companies,  each 
specialising  in  a  particular  discipline, 
resulting  in  the  present  'add-on' 
philosophy.  The  UK  MOD  has  taken  the 
view  that  the  time  is  appropriate  for 
Industry  to  adopt  a  more  coordinated 
approach.  The  Integrated  Helmet 
Technical  Demonstrator  Programme  ( IHTDP) 
is  aimed  at  encouraging  a  helmet  to  be 
designed  from  the  outset  with  wide 
ranging  capabilities,  stimulating 
Industry  to  form  consortia  to  produce  a 
truly  Integrated  Helmet.  The  paper 
discusses  the  background  to  the 
programme,  the  requirements  from  the 
operators'  point  of  view  and  describes 
the  specification  of  requirements. 

2.  THE  DEVELOPMENT  OF  THE  FLYING  HELMET 
The  modern  flying  helmet  has  evolved 
over  a  period  of  almost  50  years. 
Initially  it  was  no  more  than  a  'hard 
hat'  affording  protection  to  the  pilot 
should  he  have  to  eject  from  his 
aircraft  and  at  the  same  time  offering  a 
place  upon  which  an  oxygen  mask  and 
communication  facilities  could  be 
conveniently  attached.  The  modern 
helmet,  typified  by  the  RAF  Mk  10  ALPHA 
helmet  (Fig.l),  makes  use  of  modern 
materials  to  produce  a  light-weight, 
comfortable,  effective  element  of  his 


AEA. 

In  recent  years,  the  operational 
advantages  of  using  the  helmet  as  a 
platform  for  enhanced  vision  and  weapon 
aiming  devices  has  placed  upon  helmet 
designers  the  difficult  task  of  meeting 
the  operational  requirements  and 
producing  a  helmet  assembly  which  meets 
the  aeromedical  and  human  engineering 
aspects  of  flying  in  a  physically 
demanding  cockpit.  As  fast- jets  become 
more  agile,  the  g  loads  and  g  on-Bet 
rates  become  extreme.  Operating  for 
lengthy  periods  under  loads  approaching 
9  g  calls  for  pilots  with  a  high  degree 
of  physical  robustness.  Any  mass  on 
the  pilot's  head  must  entirely  justify 
its  existence. 

However,  it  is  generally  accepted  that 
helmet -mounted  devices  offer  a 
significant  operational  advantage  and 
could  be  the  deciding  factor  in 
determining  whether  the  pilot  reaches 
his  objective,  wins  the  engagement, 
survives  or  is  shot  down. 

Helmet -mounted  devices  have,  in 
general,  offered  a  particular 
capability.  The  next  few  sections  deal 
very  briefly  with  these  capabilities. 

2 . 1  Helmet  mounted  sights 

The  simplest  device  is  a  helmet-mounted 
sight  which  provides  a  fixed  aiming 
mark  to  the  pilot  as  a  monochromatic, 
small  field  of  view,  collimated 
reticle.  In  general  the  virtual  image 
is  formed  on  a  reflecting  patch  on  the 
clear  visor  (Fig. 2).  By  measuring  the 
pilot's  direction  of  gaze  with  a  head 
position  sensing  system  and  slewing  a 
weapon  sensor,  such  as  an  infra-red 
(IR)  seeker-head,  it  is  possible  to 
lock  the  missile  to  a  target  off  the 
aircraft  boresight.  In  this  manner  his 
firing  opportunities  are  increased  with 
obvious  operational  advantages. 

2.2  Helmet-mounted  Displays 

If  the  fixed  reticle  is  replaced  by  a 


15-2 


display  source  capable  of  providing 
cursive  symbology,  such  as  a  crt,  more 
complex  information  similar  to  that  of  a 
Head  Up  Display  (HUD)  may  be  provided. 
However,  this  imposes  more  demanding 
optical  requirements  as  the  increased 
field  of  view  required  to  present  the 
flight  data  leads  to  larger  optical 
elements  (Fig  3).  Inevitably  this 
increases  the  mass  and  a  compromise 
between  field  of  view,  exit  pupil,  eye 
relief  and  mass  must  be  made  to  meet  the 
requirements  optimally. 

Taking  this  concept  further  to  a  raster 
based  crt  system,  an  image  from  an 
electro-optical  (EO)  sensor  such  as  a 
low  light  TV  or  IR  camera  may  be 
presented  to  the  pilot  with  superimposed 
flexible  display  symbols  giving  flight 
and  mission  related  data  (Fig  4).  If 
the  EO  sensor  is  mounted  on  a  gimballed 
platform  which  is  driven  by  a  head 
position  sensing  system,  the  visually 
coupled  system  thus  formed  allows  the 
pilot  to  view  the  EO  scene  throughout  a 
field  of  regard  limited  by  the  platform 
gimbal  limits  or  his  own  ability  as  a 
contortionist.  Experience  has  shown 
that  this  type  of  image  must  be  viewed 
binocularly  to  prevent  pilot  stress, 
fatigue  and  even  nausea  due  to  binocular 
disparity. 

2.3  Night  Vision  Goggles 
Perhaps  the  most  familiar  helmet-mounted 
device  is  the  night  vision  goggle  (NVG) . 
Configured  binocularly  and  attached  to 
the  pilot's  helmet,  NVGs  provide  an 
intensified  image  of  the  view.  Adding 
approximately  800  gms  to  the  helmet, 
they  were  initially  used  in  helicopters 
where  the  lower  g  loads  were  acceptable, 
considering  the  significant  advantage  of 
being  able  to  operate  in  the  dark. 

However,  these  NVGs  were  soon  applied  to 
fast- jets,  as  illustrated  in  Fig. 5  which 
could  thus  operate  throughout  the  24  hr 
period  and  reap  the  operational  benefits 
that  this  offered.  In  this  environment, 
not  only  did  the  additional  weight 
(acting  forward  of  the  head  c  of  g) 
present  a  problem  during  normal  flight, 
where  higher  g  loadings  are  the  norm, 
the  results  of  ejecting  whilst  wearing 
the  NVGs  would  be  catastrophic.  The  UK 
MOD  were  committed  to  this  type  of 
device  for  the  SR(A)1011  programme  and 
attempts  at  reducing  the  mass  and 
forward  c  of  g  have  been  made  with  some 
success.  An  automatic  separation  device 
to  remove  them  during  the  ejection 
sequence  had  to  be  developed.  Although 
this  at  least  prevents  breaking  the 
pilot's  neck,  the  hazard  of  800  gms  of 
hardware  loose  somewhere  in  the  cockpit 
during  the  ejection  sequence  is  still  a 
cause  for  concern. 

The  above  paragraphs  summarise  the 


separate  capabilities  which  have 
enhanced  the  attributes  of  standard 
flying  helmets.  A  more  thorough 
treatment  of  these  and  other  helmet- 
mounted  de.’ices  can  be  found  in  Ref  1. 

3.  THE  REQUIREMENT  FOR  AN  INTEGRATED 
APPROACH 

Examples  of  all  the  above  devices  exist 
and  the  many  variants  providing  a 
particular  capability  bear  witness  to 
the  ingenuity  of  designers.  However, 
they  invariably  share  the  common  factor 
of  being  an  addition  to  a  helmet.  Each 
device  provides  its  own  singular  and 
unique  capability.  It  is  apparent  that 
current  and  imminent  requirements 
demand  combinations  of  the  separate 
capabilities  described  above.  For 
example,  the  combination  of  image 
intensification  for  night  use  coupled 
with  a  comprehensive  day  helmet -mounted 
display.  This  combination  not  only 
presents  the  designer  with  difficult 
and  competing  design  requirements,  it 
forces  the  weight  of  the  device  to  a 
level  which  is  incompatible  with  the 
operating  environment. 

With  this  as  a  background,  the  UK  MOD 
instigated  a  Technical  Demonstrator 
Programme  for  industry  to  produce  an 
Advanced  Integrated  Avionic  Helmet 
( AIAH)  designed  from  the  outset  with 
all  these  capabilities.  It  is 
important  at  this  point  to  look  in  some 
detail  at  the  full  reasoning  behind 
launching  this  programme. 

3 . 1  The  Team  Approach 

There  was  a  strong  belief  that  to  meet 
the  weight  and  performance  requirements 
for  fast-jet  operations,  an  integrated 
approach  was  a  prerequisite.  Helmet 
development  has  historically  been 
conducted  by  several  interrelated  but 
separate  technical  disciplines.  For 
example,  the  design  of  the  helmet  shell 
has  been  conducted  with  parameters  such 
as  mass,  strength,  comfort,  and 
penetration  resistance  uppermost  in  the 
designer's  mind. 

The  requirements  for  life  support 
functions  such  as  provision  of  oxygen, 
protection  against  blast,  NBC  and 
visual  threats  were  all  considered  as 
extras.  The  requirements  for  visual 
enhancements  have  had  no  significant 
effect  on  the  design  of  the  helmet 
shell.  Similarly,  the  optical  design 
of  helmet  display  devices  has  generally 
been  based  on  existing  helmet  shells. 
The  display  systems  manufacturers  have 
not  been  able  to  influence  helmet 
design,  only  to  make  best  use  of  the 
available  helmet  shells. 

Although  aeromedical  experts  have 
offered  their  advice  on  life  support 
and  pilot  protection  aspects,  their 
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involvement  in  HMD  programmes  would 
appear  have  been  retrospective  design 
advice  rather  than  at  the  critical 
design  concept  stage. 

It  is  believed  that  one  of  the  keys  to 
successful  helmet  system  design  is  the 
formation  of  a  design  team  consisting  of 
all  helmet  related  expertises, 
aeromedical,  optical,  display, 
materials,  etc.  and  it  was  hoped  that 
the  TDP  would  encourage  this. 

3.2  Human  Factors 

It  was  realised  that  there  was  a  lack  of 
experience  in  operating  with  these 
devices  and  their  eventual  success 
depended  upon  understanding  the  human 
factors  issues  involved.  These  factors 
can  be  separated  in  terms  of  the  pilot ' s 
psychophysical  issues  and  human 
engineering  issues. 

3.2.1  Psychophysical  Issues 
The  use  of  displays  which  provide  a 
virtual  image  present  many  effects  to 
the  viewer.  The  Head  Up  Display,  which 
evolved  from  the  gyro  gunsight,  has 
overtaken  panel  mounted  instruments  as 
the  primary  display  surface  in  military 
fast-jets,  and  was  the  first  cockpit 
display  device  to  produce  a  virtual 
image.  It  is  known  that,  although  the 
display  is  collimated,  the  pilot  cannot 
concurrently  merge  his  attention  to  the 
symbols  and  the  view  through  the 
combiner  but  effectively  switches  his 
attention  between  these  two  sources  of 
information.  With  the  addition  of  a 
sensor  image,  such  as  FLIR,  on  the  HUD 
there  are  three  images,  nominally  at 
infinity,  between  which  the  pilot  has  to 
split  his  attention.  There  is  evidence 
that  pilots  have  experienced  some 
problems  in  flying  with  information 
supplied  in  what  amounts  to  three  image 
planes,  ie  the  symbol  plane,  the  EO 
image  plane  and  the  real  world  plane. 

In  a  binocular  system,  each  eye  is 
provided  with  its  own  optical  system 
and,  strictly  speaking,  its  own 
(different)  image.  Biocular  systems 
provide  each  eye  with  the  same  image. 

The  amount  of  overlap  of  the  fields  of 
view  of  each  eye  can  be  varied  to  extend 
the  instantaneous  field  of  view  of  the 
device,  the  overlapping  area  having  the 
potential  to  provide  stereoscopic 
imagery.  However,  since  the  overlap 
area  is  provided  twice,  this  can  appear 
as  a  brighter  image  to  the  viewer  and 
the  area  immediately  adjacent  to  and 
outside  the  overlapped  region  can  give 
the  appearance  of  a  much  less  bright 
image.  This  phenomenon  is  looning,  and 
image  processing  to  minimise  the  effect 
needs  to  be  carefully  accomplished  to 
prevent  the  highly  distractive  effect 
negating  the  field  of  view  advantage 
gained  by  not  fully  overlapping  the 


fields  of  view. 

The  perception  of  distance  is  based 
upon  several  factors,  such  as  the  size 
of  an  object,  relative  brightness, 
focal  distance  of  the  eyes  and  their 
convergence.  In  synthesising  a  stereo 
image  it  is  crucial  that  these  factors 
are  taken  into  account.  Any 
inaccuracies  could  provide  false  stereo 
cuing.  For  the  eyes  to  fuse  the  two 
images  of  a  binocular  display  to  form  a 
complete  image,  the  eye/brain 
combination  must  be  given  enough 
information  to  interpret  correctly  the 
visual  data  being  provided.  Apparent 
or  real  conflicts  between  any  of  the 
factors  which  contribute  to  a 
stereoscopic  image  can  give  rise  to 
mental  and  optical  discomfort 
initially,  and  ultimately  to  higher 
workload,  and  possibly  nausea  and 
disorientation. 

The  provision  of  binocular  symbols  must 
be  carefully  considered.  Although  the 
optics  can  be  set  up  to  provide 
comfortable  viewing  at  infinity,  when 
attention  is  given  to  objects  placed 
nearer  than  infinity  the  natural 
convergence  of  the  eyes  will  cause  the 
symbols  to  appear  doubled.  The  pilot 
will  either  be  able  to  fuse  the  outside 
scene  or  the  symbols  but  not  both. 

The  above  factors  need  to  be  addressed 
thoroughly  to  provide  a  comprehensive 
data-base  from  which  future  equipment 
specifications  may  be  determined. 

3.2.2  Human  Engineering  Issues 
In  this  section  the  physical  issues 
relating  to  the  operational  use  of 
helmet -mounted  displays  is  discussed. 

Comfort  over  enduring  periods  is  a  high 
priority  for  aircrew  acceptance  of  the 
device  in  service. 

The  importance  of  the  additional  head- 
borne  mass  has  already  been  discussed. 
Even  supposing  the  mass  is  acceptable, 
the  device  must  integrate  with  the 
pilot's  aircrew  equipment  assembly 
(AEA),  must  allow  the  pilot  to  operate 
his  aircraft  in  an  unhindered  fashion. 
The  umbilical  which  connects  the  helmet 
to  the  aircraft  system  must  be 
carefully  integrated  with  the  AEA,  seat 
and  aircraft  to  allow  normal  and 
emergency  egress  from  the  cockpit  and 
not  compromise  a  safe  ejection.  The 
design  must  be  compatible  with  the  use 
of  NBC  equipment,  have  sufficient  eye 
relief  to  be  safe  and  compatible  with 
the  use  of  aircrew  spectacles.  Sizing 
of  the  helmet  and  adjustments  must 
allow  for  the  full  anthropometric  range 
of  aircrew.  The  experience  provided  by 
the  Integrated  Helmet  TDP  will  assist 
in  determining  these  factors. 
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3.3  Display  Symbology 

Considerable  experience  has  been  gained 
in  the  UK  with  a  variety  of  helmet- 
mounted  display  devices  of  limited 
capability.  A  significant  amount  of 
this  work  was  based  on  simulator  studies 
and  helicopter  flight  trials  at  DRA 
Farnborough  using  a  relatively  narrow 
field  of  view  binocular  HMD  in  a 
visually  coupled  system  configuration 
(Fig  5).  The  evidence  to  date  suggests 
that  the  design  of  symbol  formats  is  of 
vital  importance  .  The  'HUD  on  the  head 
philosophy'  is  too  simplistic  an 
approach  when  the  display  is  perpetually 
in  the  pilot's  field  of  regard. 
Particular  areas  of  concern  are  the 
amount  of  information  presented,  the 
optimum  methodology  of  moding  the 
display  for  different  phases  of  the 
flight  and  the  most  suitable  technique 
for  ensuring  the  pilot's  attitude 
awareness  at  all  times.  At  night  in 
particular,  the  pilot  must  retain  his 
awareness  of  the  aircraft  attitude  and 
his  direction  of  gaze  relative  to  the 
aircraft  axes. 

By  providing  a  flexible,  comprehensive 
symbol  generation  capability  this 
programme  will  enable  an  exhaustive 
airborne  study  of  format  design. 

3 . 4  Technology  level 

A  major  factor  driving  the  Advanced 
Integrated  Avionic  Helmet  Programme  was 
that  the  experience  gained  through  the 
programme  would  enable  the  MOD  to  have 
the  necessary  expertise  to  write 
realistic  specifications  in  the  future. 
For  this  reason,  it  was  important  that 
the  hardware  be  delivered  in  the 
shortest  possible  timescale.  It 
followed  that  the  level  of  technology 
called  for  had  to  be  current  state  of 
the  art  with  a  low  technical  risk  and  an 
associated  high  degree  of  confidence 
that  contract  would  run  to  time.  There 
was  nothing  to  be  gained  from  a 
prolonged  development  programme  with  a 
high  element  of  technical  risk.  It  was 
understood  from  the  outset  that  this 
programme  was  one  element  in  a  multi¬ 
pronged  attack  on  the  problem.  Other 
programmes  would  encourage  the 
development  of  new  technologies  which 
could  be  applied  to  future  advanced 
helmet 8. 

4.  THE  OPERATORS'  POINT  OF  VIEW 

For  many  years,  ground  attack  aircraft 
have  had  an  impressive  day  only,  good 
weather,  capability.  This  has  not  been 
matched  by  their  ability  to  operate  in 
bad  weather  or  at  night.  One  need  look 
no  further  than  the  recent  Gulf  War  to 
see  the  effectiveness  of  operating  at 
night . 

In  the  late  1960s,  the  introduction  of 
the  Head  Up  Display  allowed  the  pilot  to 


have  all  essential  flight  information 
displayed  ahead  of  him,  superimposed  on 
the  real  world.  This  allowed  him  more 
time  with  his  head  "out  of  the 
cockpit".  The  ability  to  display 
essential  flight  and  weapon  aiming 
symbology  directly  in  the  field  of  view 
of  the  pilot,  was  a  major  step  forward. 
The  reduction  in  the  cockpit  workload 
significantly  enhanced  flight  safety 
and  improved  mission  effectiveness. 
However,  as  aircraft  avionic  and  weapon 
systems  have  become  more  complex,  it 
has  become  more  difficult  for  the  pilot 
to  assimilate  all  the  essential 
information  from  the  cockpit  displays. 
Indeed,  he  cannot  take  full  advantage 
of  the  increased  capability  offered. 

The  narrow  and  fixed  field  of  view  of 
the  HUD  directly  ahead  of  the  aircraft, 
on  the  other  hand,  was  still  a  major 
limitation  of  the  system. 

With  the  arrival  of  night  vision 
sensors,  such  as  FLIR  and  NVG,  the 
limitations  of  the  HUD  fixed  field  of 
view  unduly  restricted  the  potential  of 
the  system.  The  development  of  the 
wide  angle  raster  scan  HUD,  improved 
this  position  in  that  FLIR  images  could 
be  reproduced  head-up.  Moreover,  the 
improved  FOV  significantly  enhanced 
crew  performance.  Despite  these 
improvements,  the  pilot  was  still  not 
provided  with  all-aspect  night  vision 
sensors.  A  way  forward  which  partially 
solved  the  all-aspect  issue,  was  to 
mount  NVGs  on  the  pilot's  flying 
helmet.  Whilst  helmet  mounted  NVGs 
have  proved  most  successful 
operationally,  there  are  serious 
disadvantages  of  "bolting  on"  a 
considerable  extra  mass  to  the  front  of 
the  helmet.  The  pilot  uses  the  NVG  to 
fly  at  low  level  and  to  manoeuvre  when 
the  light  levels  permit.  The  ability 
to  look  around  coupled  with  the  natural 
appearance  of  the  image,  offered  an 
acceptable  capability  despite  the  poor 
resolution  of  NVGs  relative  to  the 
FLIR.  The  advantage  of  FLIR/HUD 
combination,  with  its  high  definition 
image  ahead  of  the  aircraft,  is  that  it 
is  not  dependent  on  ambient  light 
levels.  Moreover,  the  ability  to 
indicate  potential  target  hot  spots  on 
FLIR  systems,  has  considerable 
operational  advantages. 

4 . 1  The  Requirement 

The  future  is  clear,  we  should  seek  to 
combine  all  the  visual  and 
physiological  advantages  of  FLIR  and 
NVG  within  a  fully  integrated  system. 
The  primary  function  of  the  helmet  is 
to  protect  the  crew  from  injury.  It 
should  be  robust,  light  and  comfortable 
to  wear  for  long  periods  of  time  and  be 
capable  of  displaying  flight 
information,  weapon  sighting  and  night 
vision  facilities.  The  visor  should 


also  provide  the  pilot  with  protection 
against  laser  dazzle  or  damage.  The 
Technical  Demonstrator  Programme  (TDP) 
is  one  element  of  a  two  phase  approach 
for  a  helmet  that  has  all  the  current 
safety  standards  for  crew  protection. 

The  first  phase  offers  a  full  system 
with  equipment  that  is  developed  from 
existing  technology  hardware.  The 
programme  calls  for  a  low  technical  risk 
yet  highly  capable  state  of  the  art 
solution.  The  second  phase  intends  to 
support  enha  iced  technology  for  specific 
parts  of  the  system,  where  it  is  known 
that  these  enhancements  are  not 
possible,  or  available  within  the 
timescales  of  the  first  phase. 

The  balance  of  flight,  weapon  and  night 
vision  symbology  is  critical  to  avoid 
cluttering  the  display.  The  display 
priorities  for  different  phases  of  the 
mission  and  the  need  to  select  different 
display  formats  will  be  studied  in  depth 
to  help  define  future  standards  for 
production  equipments.  The  overall 
accuracy  and  flexibility  of  current 
systems  is  below  that  required  for 
visually  coupled  systems  and  only 
suitable  for  aiming  smart  weapons  such 
LASER  Guided  Bombs  (LGB)  or  Air  to  Air 
Missiles  ( AAM) .  Performance 
improvements,  approaching  that  of  a  HUD, 
to  provide  a  display  surface  well 
registered  with  the  outside  world  is 
fundamental  in  terms  of  display 
brightness,  pointing  accuracy  and 
linearity.  This  would  permit  the  use  of 
early  generation  weapons  which  require 
precision  aiming  from  the  launch 
aircraft. 

In  summary,  the  primary  function  of  the 
helmet  is  to  protect  the  pilot  from 
injury  in  a  high  acoustic  environment. 
Prom  the  operator's  point  of  view,  the 
following  factors  should  be  given 
highest  priority:-  The  helmet  should 
have  minimum  on-head  mass  with  good 
balance,  stability  and  comfort.  The 
man-machine  interface  (MMI )  should  be 
simple  and  compatible  with  current  AEA. 
If  the  helmet  was  modular  in  design,  but 
not  role  specific,  the  design  could 
reflect  different  roles  and 
applications.  Progress  could  then  be 
made  towards  a  common  helmet  for 
different  types  of  aircraft.  Future 
techniques  of  sensor  fusion  coupled  with 
the  optimum  use  of  FLIR,  NVG  and  Radar 
or  synthetic  scenes,  will  increase  our 
operational  effectiveness.  By 
optimising  the  MMI,  mission 
effectiveness  is  enhanced  thus  further 
depriving  the  enemy  of  a  place  to  hide. 

S.  CONTRACTUAL  CONSIDERATIONS 
The  specification  of  requirements  was 
drawn  up  by  AES  with  assistance  from 
several  areas  of  expertise  within  the 
MOD  Research  Establishments,  notably 


Flight  Systems  Department  DRA 
Farnborough,  and  the  RAF  Institute  of 
Aviation  Medicine. 

The  specif ication  was  intentionally 
demanding  to  enable  MOD  to  form  an 
opinion  of  industry's  real  current 
capability,  and  it  was  fully  expected 
that  there  would  be  no  fully  compliant 
bids. 

Current  MOD  procurement  policy  dictated 
that  the  contract  should  awarded  on  the 
basis  of  an  open  competition.  The 
specification  was  therefore  sent  out  as 
part  of  the  Invitation  To  Tender  (ITT) 
to  over  20  firms  from  the  international 
avionics  industry.  The  specification 
was  accompanied  by  a  document  giving 
Special  Instructions  to  tenderers  in 
which  it  was  made  clear  to  prospective 
bidders  that  hardware  delivery  would  be 
18  months  after  receipt  of  contract. 
Tenderers  were  asked  to  include  in 
their  proposals  system  commissioning 
and  support  of  the  equipment  during  the 
flying  phases. 

The  ITT  went  out  December  1990  and  four 
months  were  allowed  for  the  respondents 
to  formulate  their  responses.  Some  of 
the  recipients  had  questions  regarding 
certain  aspects  of  the  specification, 
mostly  points  of  clarification.  In 
accordance  with  the  normal  protocol  for 
competitive  tenders  these  were  formally 
circulated  to  all  bidders  together  with 
the  official  responses. 

Four  firms  responded  to  the  ITT  and 
their  proposals  were  distributed  for 
assessment  in  May  1991. 

Due  to  the  commercially  sensitive 
nature  of  the  subject  matter  a 
nominated  team  of  assessors  was 
selected  from  the  relevant  areas  of 
MOD.  Each  member  was  given  precise 
instructions  regarding  their  terms  of 
reference  and  their  particular  areas  of 
concern.  The  technical  assessors  were 
not  made  aware  of  the  commercial 
aspects  of  the  bids,  nor  were  they 
advised  of  the  quoted  costs  or  the 
overall  value  of  the  contract. 

The  main  areas  of  concern  in 
technically  assessing  the  bids  were  the 
tenderers'  understanding  of  the 
requirements,  the  technical  compliance 
with  the  specification,  and  the 
confidence  which  could  be  placed  in  the 
supplier  meeting  his  claimed  degree  of 
compliance.  As  stated  earlier,  the 
short  timescale  for  delivery  implied 
almost  off-the-shelf  hardware.  A 
significant  element  of  the  assessment 
was  the  perceived  technical  risk  in  the 
supplier  meeting  the  claimed 
performance.  The  contract  was  not 
intended  to  be  a  developmental 
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exercise,  neither  was  it  intended  for 
pursuing  technology  research. 

A  preliminary  assessment  meeting  was 
held  to  ensure  that  all  parties 
understood  their  terms  of  reference  and 
to  confirm  the  method  of  assessment.  At 
this  stage  it  was  evident  that  most  of 
the  assessors  shared  a  common  view 
regarding  the  likely  order  of  bids, 
although  this  could  not  be  allowed  to 
effect  the  individual  conclusions  of  the 
assessment  panel  members.  Written 
reports  giving  comments  and  conclusions 
were  required  by  mid- July  1991  in 
preparation  for  the  final  technical 
assessment  meeting  in  late  July  1991. 

It  is  not  appropriate  in  this  paper  to 
discuss  the  solutions  proposed  nor  the 
reasons  for  choosing  the  eventual 
winning  bid.  However,  a  rigorous 
methodology  for  marking  the  bids  for 
technical  merit  was  developed,  based  on 
the  degree  of  compliance,  the  level  of 
importance  of  various  aspects  of 
performance,  and  a  confidence  rating 
based  on  the  assessors'  judgement  of  the 
claimed  performance. 

Similar  assessments  were  made  on  the 
relative  commercial  merits  of  the  bids 
based  on  the  commercial  proposals, 
including  costs,  although  this  activity 
was  conducted  separately  from  the 
technical  assessment  and  by  different 
panel  members. 

The  contract  was  let  in  January  1992 
with  delivery  18  months  later  in  July 
1993. 


6.  TECHNICAL  SPECIFICATION 

This  section  gives  a  cardinal  point 
specification  for  the  performance  of  the 
Integrated  Helmet.  The  specification  as 
issued  to  the  potential  tenderers  was 
much  more  detailed  and  included  other 
aspects  not  covered  here.  However,  it 
is  the  intention  that  the  details  given 
below  will  give  the  reader  some 
appreciation  of  the  level  of  performance 
required. 

6 . 1  Role  of  the  Equipment 

The  equipment  will  be  for  aircrew  flying 
fixed  wing  aircraft  in  most  operational 
roles,  with  battlefield  helicopters  as  a 
secondary  consideration.  It  will  be 
used  by  day  and  night  where  minimal 
reference  to  cockpit  displays  is 
essential.  The  sub-systems  comprising 
the  Advanced  Integrated  Avionic  Helmet 
System  are  a  light-weight  helmet, 
compatible  with  existing  UK  AEA  forming 
a  stable  platform  for  mounting  the 
avionics  and  optics,  communication  sub¬ 
system,  noise  attenuation  and  aircraft 
interfaces.  Third  generation  standard 
image  intensification  in  a  binocular 


format,  integrating  symbology  and 
imagery  from  the  video  sub-system  will 
provide  integrated  imagery  to  the  eyes. 
A  video  sub-system  interfacing  with  a 
forward  looking  infra  red  camera  (FLIR) 
and  avionics  to  provide  thermal 
imagery,  flight  and  weapon  aiming 
symbology  to  the  display  sub-system.  A 
helmet  pointing  sub-system  will 
integrate  the  helmet -mounted  avionics 
with  the  aircraft  and  weapon  systems. 

6 . 2  General 

The  system  is  to  be  optimised  for  fast- 
jet  aircrew,  particularly  with  regard 
to  total  head  mass  and  centre  of 
gravity.  No  aspect  of  the  design  shall 
increase  the  risk  of  injury  to  the 
aircrew  during  emergencies  including 
escape,  descent  and  land  or  sea 
survival.  A  fully  integrated  binocular 
display  of  imagery  is  required.  Should 
a  modular  system  be  proposed,  it  shall 
be  designed  for  single  handed  fitting 
when  wearing  NBC  AEA  and  suitable  for 
cockpit  stowage. 

6.3  Helmet  and  Communications  Sub- 
Systems 

The  helmet  shall  provide  protection 
against  blast,  high  ambient  light 
levels,  MDC  spatter,  debris  during 
ejection,  birdstrike,  etc.  Sound 
attenuation  shall  be  provide,  active 
noise  reduction  being  highly  desirable 
provided  no  mass  penalties  are  evident. 
Good  helmet  stability  to  retain  the  eye 
within  the  exit  pupil  throughout 
typical  flight  and  operational  envelope 
of  agile  faBt-jet  aircraft. 

6.4  Display  Sub-System 

6.4.1  General 

The  optical  system  shall  allow  the 
pilot  to  view  binocularly  imagery  from 
the  video  sub-system  and  Third 
Generation  Image  Intensifier  imagery 
overlaid  in  space.  A  direct  'see- 
through'  with  the  naked  eye  over  the 
whole  defined  field  of  view  shall  be 
provided. 

6.4.2  Field  of  View  ( FOV ) 

The  Image  Intensifier  images  shall  be 
circular  40°-  45°. 

The  video  image  shall  be  displayed  at 
40°  by  30°. 

The  angular  subtense  between  the  axes 
of  left  and  right  fields  of  view  must 
not  exceed  10°. 

6.4.3  Unaided  eye  FOV 

At  least  160°  horizontal  x  80° 
vertical.  Individual  visual 
obstructions  shall  be  less  than  2° 
subtended  angle  with  no  obstructions 
within  the  display  fields  of  view. 

6.4.4  Linearity 

Any  EO  image  of  an  object  at  infinity 
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■hall  be  within  D  degrees  of  its  true 
angular  position  where  D  =  0.1  x  R  where 
R  is  the  true  angular  subtense. 

6.4.5  Veiling  Glare 

2%  on  axis.  Haze  less  than  0.5% 

6.4.6  Alignment 

Misalignment  between  any  image  at  the 
centre  of  the  FOV  and  the  infinity 
outside  world  shall  be  less  than  1.0 
mrad. 

6.4.7  Eye  Relief  and  Exit  Pupil 

Eye  relief:  30  mm  min.  Exit  pupil:  15 
mm  min. 

6.4.8  Virtual  Image  Distance 

The  image  viewed  through  the  display 
optics  shall  be  focused  between  500  m 
and  true  infinity.  The  system  shall 
provide  a  means  of  minimising  any 
adverse  optical  effects  when  the 
aircrew's  eyes  converge  to  focus  on 
directly  viewed  near  objects. 

6.4.9  Transmission  and  Reflectance 
Minimum  photopic  transmission  of  70%  for 
directly  viewed  objects.  Display  optics 
shall  have  a  minimum  transmission  of  25% 
between  540  and  550  nm. 

6.4.10  Magnification 

System  magnification  1.0  ±  0.04  with 
each  monocular  assembly  matched  to 
within  t  0.02. 

6.4.11  Resolution 

At  least  0.8  cy/mrad  on  axis  for  full 
FOV.  At  least  0.73  cy/mrad  10°  off 
axis. 

At  least  0.73  cy/mrad  with  pupil 
displaced  5  mm  from  centre  of  optical 
axis. 

6.4.12  Brightness  Gain 

Not  less  than  2000  cd/m2  per  cd/m2  with  a 
difference  between  monoculars  of  less 
than  10%. 

6.4.13  Image  Intensification 

The  image  intensification  shall  allow 
the  pilot  to  use  normal  visual  flying 
techniques  in  light  conditions  down  to 
overcast  starlight,  5  x  10 4  lux  and 
shall  have  a  third  generation  spectral 
response.  The  tube  specification 
proposed  shall  identify  significant 
improvements  over  existing 
specifications  in  the  areas  of 
blemishes,  fixed  pattern  noise, 
scintillation,  gross  and  shear 
distortion  (if  a  fibre  optic  twister  is 
used) . 

6.5  Video  Sub-System 

A  video  sub-system  based  on  a  high 
performance  miniature  crt  or  display 
device  of  similar  performance  to  provide 
imagery  from  a  FLIR  thermal  imager, 
weapon  system  data  and  flight  symbology 
to  the  display  sub-system. 


6.5.1  Electrical  Interfaces 

The  system  shall  provide  an  interface 
with  the  aircraft  avionics  using  a  MIL 
STD  1553B  interface.  It  shall  also 
provide  an  interface  with  the  Helmet 
Pointing  sub-system. 

6.5.2  Video  Interface 

625  line  50  Hz  CCIR  standard. 

Additional  interfaces  to  allow  growth 
potential  for  future  high  definition 
video  standards  and  planned  FLIR 
improvements  are  desirable.  The  system 
shall  interface  with  the  SR(A)  1010 
fixed  FLIR.  As  the  helmet  traverses 
the  FLIR  field  of  regard  the  system 
shall  slew  the  displayed  FLIR  image  to 
maintain  registration  of  the  FLIR  image 
with  the  outside  world  to  within  1 
mrad.  The  image  shall  be  stabilised 
with  reference  to  airframe  to 
compensate  for  both  deliberate  and 
inadvertent  head  motion  due  to 
vibration/ turbulence. 

6.5.3  Brightness 

Automatic  brightness  control  shall  be 
provided  together  with  a  manual 
override.  The  symbology  shall  be 
legible  when  viewed  against  a 
background  luminance  of  35000  cd/m2. 

6.5.4  Symbology 

The  system  shall  provide  simple  flight 
data,  weapon  status  and  aiming 
symbology  configured  initially  in  a 
typical  HUD  format.  The  provision  of 
information  shall  be  software 
controlled  allowing  flexibility  to 
change  content  and  format. .  A  means  of 
quickly  decluttering  the  display  in 
flight  shall  be  provided. 

6.5.5  Pilot  Controls 

The  pilot  shall  be  provided  with 
adequate  controls  to  operate  the  system 
in  flight. 

6.5.6  Linearity 

The  video  system  shall  provide  all 
adjustments  required  to  ensure  that  the 
video  images  as  seen  at  the  eyes  are 
aligned  with  real  objects  at  true 
infinity  to  within  1  mrad. 

6.5.7  Resolution  (with  CCIR  TV  input) 
Over  the  full  exit  pupil  and  with  the 
image  rotated  through  0°,  45°,  and  90° 
the  system  shall  display  a  bar  pattern 
of  frequency  8  MHz  at  full  contrast 
against  a  background  luminance  of  5000 
cd/m2  (also  4  MHz  at  20%  contrast). 

Similarly,  with  a  background  luminance 
of  less  than  0.1  cd/m2  a  full  contrast 
bar  pattern  of  6  MHz  shall  be  displayed 
(also  3  MHz  at  20%  contrast). 

6.5.8  Contrast 

The  system  shall  be  capable  of 
maintaining  a  minimum  contrast  of  1.2 
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against  a  real  world  luminance  of  30,000 
cd/mJ. 

6.5.9  Symbology 

The  system  shall  be  capable  of 
displaying  symbols  of  line  width  1  mrad 
approximately  to  a  positional  accuracy 
consistent  with  the  linearity  and 
magnification  requirements. 

6.5.10  Grey  Scale  Resolution 

The  video  sub-system  shall  be  capable  of 
displaying  eight  shades  of  grey 
including  black,  where  shades  are 
related  by  a  level  factor  of  1.4. 

6.6  Helmet  Pointing  Sub-System 

Components  shall  not  hinder  normal 
aircrew  tasks  or  mobility.  The  system 
shall  be  capable  of  functioning  with 
inputs  from  two  helmets.  Outputs  of 
position  and  angle  shall  be  provided  in 
a  recognisable  and  standard  coordinate 
system  referenced  to  aircraft  axes. 

6.6.1  Minimum  Head  Motion  Box 
(referenced  to  aircraft  axes): 

Translation  Rotation 

x  360  mm  Pitch  ±80° 

y  300  mm  Roll  ±70° 

z  200  mm  Yaw  ±160° 

6.6.2  Linear  Resolution 

<  t  1  mm 

6.6.3  Angular  Accuracy 

Within  the  ±30°  forward  cone  <  1  mrad 

Within  the  ±80°  forward  cone  <  5  mrad 

Elsewhere  in  head  motion  box  <  20  mrad 

6.6.4  Angular  Resolution 

<  0.5  mrad 

6.6.5  Minimum  Update  Rate 
50  Hz 

6.6.6  Boresighting 

A  means  of  in  flight  boresighting  the 
helmet  optical  sightline  with  an 
aircraft  datum  is  required  to  an 
accuracy  of  0.5  mrad. 

6.7  Design  and  Construction 

6.7.1  Mass 

Maximum  permitted  head  mounted  mass  is 
2.0  Kg  (excluding  oxygen  mask) 

Desirable  mass  1.4  Kg  (excluding  oxygen 
mask) 

6.7.2  Centre  of  Gravity  (CG) 

±y  direction  As  per  unadorned  head, 

tx  direction  As  far  rearwards  as 

possible  within  the 
bounds  of  the  natural 
CG  and  a  point  above 
the  rotation  point  of 
the  head. 

As  close  as  possible 
to  the  natural  CG  but 
not  exceeding  40  mm 


above  natural  CG,  and 
highly  desirable  that 
it  be  as  low  as 
possible. 

6.7.3  Maximum  Dimension  Profile 

No  part  of  the  helmet  shall  protrude 
more  than  200  mm  upwards  or  225  mm 
rearwards,  from  the  eye  position.  The 
helmet  shall  not  be  wider  than  280  mm. 
Aircrew  mobility  when  wearing  full  AEA 
shall  not  be  impeded. 

6.7.4  Connectors  and  Cables 

Any  connection  to  the  head  mounted 
systems  shall  be  accessible  to  the 
aircrew,  provide  single  handed 
connection  and  disconnection  with 
protection  against  inadvertent 
disconnection  and  shall  separate 
automatically  on  ejection.  Electrical 
discharge  protection  shall  be  provided. 
Any  cables  shall  not  inhibit  aircrew 
mobility  in  normal  or  emergency  tasks. 

6.7.5  Environmental 

The  equipment  design  should  be 
consistent  with  operating  in  a  fast-jet 
environment . 

Note:  Standard  environmental  tests  and 
performance  requirements  were  given  in 
the  specification  covering  temperature, 
pressure,  humidity,  vibration,  shock 
etc. 

6.7.6  Availability 

The  equipment  shall  be  designed  for  a 
service  life  of  5  years. 

6.7.7  Reliability 

A  mean  time  between  defects  of  1000 
hours  shall  be  achieved.  Flight  safety 
critical  failures  shall  not  exceed  1  in 
106  flying  hours. 

6.7.8  Maintainability 

The  mean  active  repair  time  at  first 
line  shall  not  exceed  30  min  and  it 
shall  be  possible  for  90%  of  all 
repairs  to  be  completed  within  1  hour. 

6.7.9  Contractor  Support 

For  a  period  of  18  months  from  delivery 
of  the  equipment  the  contractor  shall 
provide  engineering  support,  repair  and 
maintenance  including  the  provision  of 
spares,  support  any  software  changes, 
updates  or  reprogramming  required 
either  due  to  failures  or  as  part  of  an 
agreed  programme  of  software 
development . 

6.7.10  Design  Approval 

Limited  type  approval  testing  shall 
cover  the  following  aspects: 

Crash  safety/shock  including 
impact  and  crash  deceleration 
tests 
Vibration 


±z  direction 
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Temperature/pressure 
Explosive  decompression 
EMC 

Compass  safe  distance 
Performance  in  sustained 
acceleration 
Air  blast  to  450  knots 
Ejection  ramp  tests 
AEA  compatibility 
Emergency  egress 

7.  CONCLUDING  STATEMENTS 

It  will  be  seen  from  the  above  that  the 
specification  was  quite  demanding,  the 
main  features  making  it  so  being  the 
large  binocular  FOV,  the  large  exit 
pupil,  the  requirement  for  image 
intensification  and  concurrent  sensor 
imagery  and,  last  but  not  least,  the  all 
up  mass  of  2.0  Kg  maximum.  In  addition 
to  these  physical  requirements,  the 
optical/display  requirements  called  for 
a  somewhat  better  performance  than 
current  systems'  known  capabilities. 

There  were  good  reasons  for  demanding 
this  performance,  apart  from  being  able 
to  bench  mark  industry's  current 
capabilities.  The  aeromedical 
fraternity  are  only  too  familiar  with 
upper  spine  and  neck  injuries  to  pilots 
wearing  'unadorned'  helmets  in  modern 
fast-jets  caused  by  the  high  positive 
accelerations  (gj  and  high  onset  rates. 
A  strong  emphasis  was  therefore  given  to 
all  up  mass,  even  for  limited  trials 
flying. 

Considerable  experience  has  been  gained 
flying  NVGs.  In  enhancing  the  FOV  from 
the  30°  of  earlier  versions  to  the  40° 
of  more  recent  types,  significant 
improvements  were  gained  both  in 
performance  and  subjective  ease  of  use. 
It  was  considered  in  the  opinion  of  the 
technical  contributors  to  the 
specification  that  to  reduce  the  FOV 
available  would  be  a  retrograde  step  and 
unacceptable  to  pilots  used  to  a  wide 
FOV  device. 

The  brightness  and  contrast  requirements 
for  all  modes  of  operation,  ie  day  and 
night  flying,  demanded  high  performance 
devices  and  efficient  optical  design. 

The  high  transmissibility  requirement 
ensures  that  the  normal  view  to  the 
outside  world  would  not  be  degraded  in 
the  interest  of  display  legibility 
{brightness/contrast ) . 

The  experience  gained  in  conducting  this 
programme  will  provide  the  UK  MOD  with 
many  benefits.  Procuring  the  helmet 
will  provide  a  measure  of  industry's 
current  capability,  flying  the  AIAH  will 
be  beneficial  in  building  expertise  in 
the  human  engineering  and  human  factors 
associated  with  these  devices,  and 
operating  with  the  helmet  in  a  realistic 
environment  will  enable  a  raft  of 


knowledge  to  be  built  up  which  will 
help  establish  operational  techniques 
for  the  Integrated  Helmet  equipped 
pilots  of  the  future. 
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I  INTRODUCTION 

II  Traditional  Roles 

The  primary  role  of  aircrew  visors  has  always  been  to 
provide  facial  protection  in  the  event  of  ejection,  bird 
strike  or  other  hazards.  It  most  achieve  this  with 
negligible  degratation  to  normal  vision,  Le.  very  high 
transmission,  minimal  colouration,  minimal  optical 
distortion.  The  material  used  is  inevitably 
polycarbonate,  which  combines  high  mechanical 
strength  with  good  optical  qualities. 

A  secondary  visor  role  has  traditionally  been  to  provide 
optical  protection  against  solar  glare.  This  is  achieved  by 
providing  a  dyed  visor  which  gives  relatively  low 
transmission  (10-20%)  uniformly  across  the  visible 
bond.  These  sun  visors  must  provide  excellent  protection 
against  ultra-violet  exposure  and  to  a  lesser  extent 
against  near  Infra  Red  exposure.  Similar  requirements  on 
mechanical  and  ophthalmic  performance  apply.  The  sun 
visor  can  be  used  instead  of  a  clear  visor  or  in 
conjunction  with  a  clear  visor.  The  twin  visor  system 
ensures  that  the  mechanical  protection  of  the  dear  visor 
is  always  present  whilst  the  deployment  of  the  sun-visor 
is  optional,  at  the  expense  of  additional  weight.  This  is 
the  approach  currently  adopted  by  the  U.K. 

12  Optical  Threats 

Mote  recently  additional  protective  requirements  have 
evolved,  which  may  also  need  to  be  incorporated  into  the 
visor.  These  are  the  threats  to  the  pilot’s  vision  posed  by 
either  lasers  or  the  flash  from  a  nuclear  explosion. 

The  threat  from  lasers  may  be  at  Dazzle  or  Damage 
levels.  Dazzle  threats  are  typically  c.w.  lasers  at 
relatively  low  powers,  but  which  would  cause  temporary 
flash-blindness  and  would  preclude  viewing  in  the 
source  direction. 

These  threats  can  typically  arise  over  a  wide  band  of 
wavelengths  and  active  measures  to  counter  them  must 
respond  in  milliseconds. 

Damage  threats  are  posed  by  high  power  pulsed  lasers 
and  may  cause  permanent  blindness  in  both  eyes.  Active 
measures  to  counter  these  must  respond  within  a  fraction 
of  the  timescale  of  the  pulse  (typically  nanoseconds). 
These  threats  me  primarily  limited  to  a  number  of 
specific  wavelengths  corresponding  to  high  effiriency 
pulsed  lasers.  They  can  therefore  be  countered  by 
permanent  fixed-line  rejection  filters  at  the  relevant 
wavelengths. 

Nuclear  flash  threats  occur  at  ranges  where  the 
explosion  is  survivabte  but  the  intensity  of  the  visible 
component  of  the  flash  is  sufficient  to  pose  hazards 
ranging  from  severe  flash-blindness  to  permanent  eye 
damage.  Prelection  must  be  provided  across  the  entire 
visible  band,  typically  in  a  timescale  of  milliseconds. 


13  Why  Visors? 

Ocular  protection  against  these  threats  could  be  provided 
in  a  number  of  formats,  for  example  spectacles  or 
goggles  or  even  integrated  into  the  canopy.  However,  all 
such  protection  will  carry  a  penalty  in  terms  of  normal 
visual  transmission. 

This  will  be  discussed  in  more  detail  later.  Where  there 
is  a  permanent  transmission  loss  associated  with  the 
protective  device  it  is  vital  that  a  "look-past"  capability 
be  retained  so  the  pilot  can  assess  the  relative 
importance  of  protection  or  transmission  at  any  given 
time  versus  other  threats  which  may  be  present  and 
deploy  the  device  accordingly. 

The  optimum  formal  for  this  protection  is  therefore  the 
aircrew  visor,  which  can  be  rapidly  and  safely  moved  in 
and  out  of  the  field  of  virion. 

L4  HMD  Combiners 

Finally,  a  further  role  for  the  modem  aircrew  visor,  is  to 
function  as  the  combiner  in  a  Helmet  Mounted  Display 
system.  In  any  such  system  the  combiner  must  be 
located  in  front  of  the  eyes  and  fixed  to  the  helmet 
Time  are  systems  with  separate  combiners  located 
inside  the  visor,  but  these  pose  serious  weight  and  safety 
problems.  The  most  practical  solution  is  to  use  the  visor 
itself  as  the  system  combiner  by  providing  suitable 
spectrally  selective  filters  at  the  visor  surface. 

13  Multi-Function  Visors 
The  role  of  visors  in  future  cockpits  is  therefore  greatly 
expanded.  Each  of  these  requirements  cannot  be  met 
with  a  separate  visor  since  the  overall  weight  of  the 
helmet  assembly  is  also  critical,  especially  in  higjh-g 
aircraft.  Ideally  all  of  these  requirements  would  be  met 
with  a  single  visor,  or  at  most  a  pair  of  visors.  This  gives 
rise  to  the  concept  of  the  Multi-Function  Visor  (MFV). 

A  single  visor  that  meets  the  full  range  of  all  of  these 
requirements,  with  good  transmission,  is  probably  not  a 
feasible  concept.  This  is  especially  true  for  the 
protection  against  numerous  laser  lines  in  the  visible 
band.  However,  what  can  be  achieved  are  visors 
offering  combinations  of  these  requirements  whilst  still 
retaining  an  acceptably  high  transmission.  The  concept 
therefore  is  a  suite  of  Multi-Function  Visas,  all  meeting 
some  basic  requirements  and  offering  different 
protective  combinations  on  a  mission-selective  basis. 

This  presentation  will  very  briefly  survey  the 
technologies  available  to  meet  these  requirements, 
highlighting  the  effects  on  transmission.  The  application 
of  these  technologies  to  visors  will  be  assessed  and  some 
of  the  work  to  date  at  Pilkington  Optronics  will  be 
reported. 
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Finally  the  combination  of  these  technologies  to  meet 
die  Multi-Function  Visor  requirements  will  be 
addressed. 

A  summary  of  the  requirements  and  constraints  for 
Multi-Function  Visas  is  shown  in  Figure  1. 


2  TECHNOLOGIES 

2.1  Moulding  Technology 
Injection  moulded  polycarbonate  visas  are  currently  the 
oily  way  of  achieving  the  necessary  mechanical 
performance  with  good  optical  quality. 

Polycarbonate  is  inherently  soft  so  the  exposed  surfaces 
must  be  protected  with  an  anti-scratch  coating. 

All  additives  (e.g.  dyes)  a  surface  processes  must  be 
such  that  they  do  not  reduce  the  mechanical  performance 
of  the  polycarbonate. 

The  optical/ophthalmic  quality  of  the  visa  is  primarily 
determined  by  the  design  and  surface  finish  of  the 
moulding  tool  and  die  control  of  the  moulding  process. 
The  visa  strength  is  enhanced  if  drilling  and  edging 
operations  are  avoided  by  moulding  to  final  shape. 

2Jt  Absorption  Technologies 
The  transmission  characteristics  of  an  optical  component 
can  be  modified  by  incorporating  selective  absorbing 
materials  within  it  These  are  two  major  classes  of 
absorbent  materials:  absorbing  glasses  and  organic  dyes. 
Glass  materials  are  not  suitable  fa  use  in  visors,  on 
mechanical  and  weight  grounds. 

Dyes  (primarily  organic  dyes)  can  be  incorporated  into 
the  polycarbonate  prior  to  moulding,  a  in  some  cases 
can  be  imbibed  into  moulded  components.  The  spectral 
properties  of  known  dyes  vary  from  broad-band  to 
narrow-band,  across  the  U.V.  and  visible  spectrum.  The 
available  choice  diminishes  in  the  NIR. 

To  survive  moulding  in  polycarbonate,  the  dyes  must 
withstand  high  temperatures.  Most  organic  dyes  show 
some  degree  of  solar  degradation  on  long  term  exposure. 
Many  organic  dyes  saturate  if  illuminated  intensely  at 
one  wavelength  (e.g.  with  a  laser). 

There  are  sane  very  narrow  band  absorbing  dyes 
available,  but  even  these  have  leading  and  trailing 
absorption  edges  which  significantly  reduce  the  overall 
visual  band  transmittance. 

23  Reflection  Technologies 
Specific  wavelengths  or  wavelength  bands,  can  be 
selectively  reflected  by  the  use  of  interference  films. 
Transmission  levels  at  given  wavelengths  can  be 
reduced  to  0.01%  (OD4)  or  lower.  The  interference 
films  can  be  in  the  form  of  holographic  filters  a 
multi-layer  vacuum  coatings.  These  can  be  designed  to 
give  very  sharp  spectral  characteristics  and  to  operate  at 
any  given  wavelength.  Combinations  of  such  filters  can 
reflect  multiple  wavelengths  across  the  spectrum.  Until 
recently  it  has  been  very  difficult  to  apply  these 
technologies  to  large,  highly  curved,  polycarbonate 
visors. 

The  traditional  holographic  material,  Dichromated 
Gelatin  or  DCG  is  degraded  by  water  vapour  and  must 
be  protected  by  glass.  However,  new  holographic 
materials  are  more  durable  and  have  been  successfully 
applied  to  large  polycarbonate  substrates. 

Traditional  techniques  in  vacuum  coating  are  unable  to 


deposit  complex  multi-layer  coatings  with  sufficient 
uniformity  across  large,  curved  visas.  Recent 
developments,  notably  at  Pillrington  Optronics,  have 
overcome  this.  Similarly,  mismatch  problems  between 
the  visa  material  and  typical  coating  materials  have 
ban  overcome  to  ensure  good  environmental  stability. 

There  is  still  one  very  fundamental  limitation  on  the 
application  of  interference  technologies  to  provide 
protective  reflection.  The  spectral  performance  of  all 
interference  filters  is  dependat  on  the  incidence  angle 
of  the  light  At  high  angles  the  reflection  is  shifted  to 
shorter  wavelengths.  To  maintain  reflectivity  at  a 
specific  wavelength,  over  a  wide  range  of  incidence 
angles,  the  spectral  width  of  the  reflection  filter  must  be 
extended.  This  reduces  the  transmission. 

Fa  optical  threats  in  the  Red  a  NIR  the  filter  can  be 
extended  without  significant  loss  of  visual  transmission, 
but  fa  threats  within  the  visual  band  the  loss  in 
transmission  can  be  severe. 

The  angular  coverage  required  to  provide  good  optical 
protection  is  therefore  a  critical  parameter  in  the  design 
of  protective  interference  filters.  Pillrington  Optronics 
have  devised  a  cries  of  computer  models  to  accurately 
determine  this  lor  a  wide  range  of  devices,  including 
visors. 

The  angular  coverage  required  depends  on  the  shape  and 
curvature  of  the  substrate  or  visa.  Very  highly  curved 
visas,  matched  to  the  eye  rotation  curve,  require  the 
minimum  angular  coverage  and  therefore  could  have  the 
highest  transmission.  Such  visors  are,  however, 
impractical.  Using  holography,  a  technique  can  be 
adopted  where  the  construction  geometry  is  designed  so 
that  the  effective  shape  of  the  filter  does  not  follow  the 
shape  of  the  substrate  it  is  deposited  on.  This  is  known 
as  Non-Conformal  Holography.  This  offers  the 
possibility  of  good  angular  coverage  and  good 
transmission  on  a  traditional  visa  shape  and  is  currently 
an  active  area  of  development. 

2.4  Active  Technologies 

The  technologies  described  so  far  have  all  been  passive, 
i.e.  their  spectral  performance  is  fixed.  There  are  a  wide 
range  of  active  filter  technologies,  where  the  spectral 
characteristics  are  modified  in  response  to  the  incident 
radiation.  It  is  not  possible  to  do  justice  to  this  range  in 
such  a  brief  paper.  However,  most  of  these  technologies 
are  in  the  form  of  devices,  with  requirements  fa 
detectors,  applied  voltages,  associated  electronics  or 
focussing  optics.  In  general,  they  offer  a  limited 
aperture  a  limited  field  of  view  and  would  probably  be 
used  in  a  goggle  format.  The  exception  to  this  is 
photochromic  technology,  which  requires  ro  external 
components  and  offers  excellent  prospects  for 
incorporation  into  multi-function  visors.  I  will  therefore 
concentrate  solely  on  photochromic  dyes. 

Photochromic  dyes  darken  under  appropriate 
illumination  and  are  well  known  in  the  form  of 
sun-glasses.  Pillrington  are  world  leaders  in  the 
development  of  photochromic  technology  and  the  latest 
generation  of  dyes  offer  exceptional  switching  speeds, 
good  optical  densities  and  good  coverage  across  the 
visual  spectrum. 

The  dyes  function  by  absorbing  near  u.v.  and  blue  light 
which  modifies  die  molecular  structure  to  create 
flfUtirirmnl  absorption  bands  in  the  visible  spectrum.  The 
dyes  can  respond  in  pico  seconds  (under  pulsed  laser 
illumination)  and  fa  millisecond  illumination  pulses  the 
dye  response  exaedy  follows  the  incident  pulse-shape. 
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These  dyes  can  be  used  for  both  active  sun-visors  and 
nuclear  flash  protection.  The  amount  of  u.vTblue  light 
transmitted  through  aircraft  canopies  is  sufficient  to 
trigger  the  photochromic  response. 

The  dyes  can  be  incorporated  into  a  range  of  polymer 
host  materials,  but  unfortunately  not  polycarbonate. 

Utilising  these  technologies,  a  number  of  optimised 
solutions  to  die  Multi  Function  Visor  Problem  can  be 
produced,  each  depending  largely  upon  the  priority 
given  to  die  various  functions  the  visor  must  perform. 


3  MULTI-FUNCTION  VISOR  DESIGNS 

3.1  MFV 1  •  Solar/Laser  Protection  (low 
transmission) 

One  obvious  multifunction  visor  approach  is  to  use 
combinations  of  narrow  band  laser  absorbing  dyes  in  a 
visor,  tailored  to  ensure  the  overall  visual  transmittance 
is  at  the  10-20°  solar  protection  level.  This  sun-visor 
then  provides  laser  protection  at  a  number  of  discrete 
lines.  With  the  dyes  incorporated  into  polycarbonate 
these  visors  also  meet  all  the  ophthalmic  and  mechanical 
requirements. 

This  approach  has  been  successfully  adopted  to  provide 
2-line  and  3-line  laser  protection  visors.  With  the 
appropriate  selection  of  dyes,  the  colouration  can  be 
reasonably  neutral,  with  excellent  u.vjblue  blocking  and 
good  NIR  attenuation.  There  are  limits  on  die  number  of 
laser  lines  that  can  be  blocked  whilst  maintaining 
adequate  transmission. 

3  2  MFV  2  -  Soiar/Nndear  Flash/Laser 
Protection  (moderate  transmission) 

The  MFV  2_  design  is  only  suitable  for  daytime  laser 
protection.  For  night  time  use  die  laser  protection  must 
be  provided  by  combinations  of  vacuum  coatings, 
non-ionformal  holograms  (and  possibly  some 
narrow-band  dyes).  The  optimum  designs  must  provide 
high  transmission,  typically  in  excess  of  30%.  Inis  can 
be  achieved  with  coverage  over  perhaps  3-7  laser  lines, 
depending  on  the  precise  choice  of  wavelengths. 

Photochromic  dyes  can  be  incorporated  into  the  visor  to 
provide  solar  glare  protection  fra  daytime  use.  With  the 
appropriate  mix  and  concentrations  of  dye,  the  visor 
transmission  will  respond  to  ambient  illumination  levels 
and  provide  a  more  adaptable  form  of  protection  than  the 
current  sun-visors. 


The  intention  is  that  the  same  photochromic  dyes  will 
also  provide  protection  against  nuclear  flash.  There  will 
need  to  be  a  compromise  between  the  optimum  dye  mix 
fra  nuclear  flash  protection  and  that  fra  solar  protection 
and  the  optimum  concentrations  are  currently  being 
investigated. 

The  transmission  value  fra  MFV  2  depends  on  the 
precise  combinations  of  laser  lines.  Protection  against 
lasers  at  the  centre  of  the  photopic  eye  response,  or  on 
the  shot  wavelength  side  of  it  (fra  interference  filters), 
results  in  a  greater  loss  of  visual  transmission.  In 
combining  protective  filters,  care  must  also  be  taken  to 
ensure  that  acceptable  colour  balance  is  maintained.  In 
particular,  it  is  vital  that  large  sections  of  the  visible 
spectrum  are  not  blocked  (fra  example  "blue”)  since  this 
win  degrade  colour  discrimination. 

Another  vital  parameter  is  display  compatibility, 
especially  with  narrow  band  phosphors  in  Head-Up  and 
Head-Down  displays  and  with  various  shades  of  panel 
display  and  warning  lights. 


In  reality,  the  full  range  of  potential  laser  threats  in  the 
visible  band  cannot  be  covered  on  a  single  visor  whilst 
maintaining  adequate  visual  transmittance  and  colour 
balance.  There  will  need  to  be  a  suite  of  MFV  2  visors 
offering  different  combinations  of  laser  protection,  all 
with  photochromic  solar  and  nuclear  flash  protection,  all 
with  U.V.  and  NIR  protection.  The  choice  of 
appropriate  visor  will  be  made  on  a  mission  by  mission 
basis. 

3J  MFV  3  •  HMD  Combiner/Mechanical  Protection 
The  one  aspect  not  covered  in  the  MFV  2  visor  is  the 
HMD  combiner.  This  will  typically  be  a  very  narrow 
band,  non-conframal  holographic  filter  providing  high 
reflectivity  over  the  limited  range  of  angles  required  for 
the  HMD  system.  In  addition  it  will  almost  certainly 
need  to  supply  some  of  the  aberration  correction  fra  the 
overall  system  and  will  therefore  be  a  complex  hologram 
to  construct 

In  principle,  this  could  be  inducted  in  the  MFV  2  visor 
with  little  degradation  in  overall  transmittance. 
However,  the  MFV  2  visor  will  have  limited  visual 
transmittance  and  it  is  regraded  as  vital  that  the  pilot 
retains  a  "look-past"  capability  by  pushing  this  visor  up, 
out  the  field  of  view.  This  would  not  be  possible  if  it 
functions  as  the  HMD  combiner. 

The  proposed  solution  therefore  requires  a  twin  visor 
system.  An  inner  clear  visor  with  the  HMD  combiner, 
which  is  permanently  down  and  an  outer  Multi-Function 
Visor  whose  deployment  is  controlled  by  the  pilot  The 
inner  visor  will  have  very  high  visual  transmittance  and 
will  provide  full  mechanical  protection. 

Hie  outer  visor  will  have  limited  visual  transmittance 
(depending  on  the  multi-functions)  and  need  not  provide 
full  mechanical  protection.  The  weight  of  the  outer  visor 
can  therefore  be  reduced. 


4  SUMMARY 

A  multitude  of  new  roles  are  required  for  future  aircrew 
visors.  Large  numbers  of  visors  on  a  single  helmet  is  not 
an  acceptable  solution.  There  is  therefore  a  requirement 
fra  Multi-Function  Visors  which  perform  numerous 
rotes. 

Many  of  the  optical  protective  functions  cany  an 
unavoidable  loss  of  visual  transmittance.  When  these 
are  combined,  the  overall  visual  transmittance  may 
typically  be  40%  or  lower.  These  functions  therefore 
need  to  be  combined  on  an  outer  Multi-Function  Visor 
which  the  pilot  can  deploy  at  will.  A  permanent  inner 
clear  visor  will  provide  all  the  mechanical  protection  and 
will  also  host  the  HMD  combiner. 

A  suite  of  outer  MFV’s  may  be  required  to  cover  all  the 
combinations  of  laser  threats  in  the  visible,  with  the 
appropriate  visor  being  selected  fra  each  mission. 
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1.  Introduction 

After  almost  three  scores  of  years  of  basic  and  applied 
research,  the  field  of  speech  processing  is,  at  present,  under¬ 
going  a  rapid  growth  in  terms  of  both  performance  and 
applications  and  this  is  fuelled  by  the  advances  being  made 
in  the  areas  of  microelectronics,  computation  and  algorithm 
design.  Speech  processing  relates  to  three  aspects  of  voice 
communications  [1]: 

-Speech  Coding  and  transmission  which  is  mainly  con¬ 
cerned  with  man-to-man  voice  communication. 

-Speech  Synthesis  which  deals  with  machine-to-man 

communication. 

-Speech  Recognition  which  is  related  to  man-to- 

machine  communication. 

The  paper  first  discus¬ 

ses  the  use  of  voice  for  civil  and  military  communications 
and  considers  its  advantages  and  disadvantages  including  the 
effects  of  environmental  factors  such  as  acoustic  and  electri¬ 
cal  noise  and  interference  and  propagation.  The  structure  of 
the  existing  NATO  communications  network  and  the  evolv¬ 
ing  Integrated  Services  Digital  Network  (ISDN)  concept  is 
briefly  reviewed  to  show  how  they  meet  the  present  and 
future  requirements.  It  is  concluded  that  speech  coding  at 
low-bit  rates  is  a  growing  need  for  transmitting  speech  mes¬ 
sages  with  a  high  level  of  security  and  reliability  over 
capacity  limited  channels  and  for  memory-efficient  systems 
for  voice  storage,  voice  response,  and  voice  mail  etc.  Fur¬ 
thermore  it  is  pointed  out  that  the  low-bit  rate  speech  coding 
can  ease  the  transition  to  sharer!  channels  for  voice  and  data 
and  can  readily  adopt  voice  messages  for  packet  switching. 

The  paper  then  deals  with  the  fundamental  subject  of  speech 
coding  and  compression.  Recent  advances  in  techniques  and 
algorithms  for  speech  coding  now  permit  high  quality  voice 
reproduction  at  remarkably  low  bit  rates.  The  advent  of 
powerful  single-chip  signal  processors  has  made  it  cost  ef¬ 
fective  to  implement  these  new  and  sophisticated  speech 
coding  algorithms  for  many  important  applications  in  voice 
communication  and  storage. 

The  subject  of  speech  synthesis  is  next  treated  where  the 
principal  objective  is  to  produce  natural  quality  synthetic 
speech  from  unrestricted  text  input.  Useful  applications  of 
speech  synthesis  include  announcement  machines  (e.g. 
weather,  tirae)computer  answer  back  (voice  messages, 
prompts),  information  retrieval  from  databases(stock  price 
quotations,  bank  balances)  reading  aids  for  the  blind,  and 
speaking  aids  for  the  vocally  handicapped. 

Probably  the  most  intractable  of  all  the  speech  processing 
techniques  which  is  last  discussed  in  the  paper  is  speech 
recognition  where  the  ultimate  objective  is  to  produce  a 


machine  which  would  understand  conversational  speech 
with  unrestricted  vocabulary,  from  essentially  any  talker. 
Algorithms  for  speech  recognition  can  be  characterized 
broadly  as  pattern  recognition  approaches  and  acoustic 
phonetic  approaches.  To  date,  the  greatest  degree  of  success 
in  speech  recognition  has  been  obtained  using  pattern  recog¬ 
nition  paradigms.  It  is  for  this  reason  that  the  paper  is  con¬ 
cerned  primarily  with  this  technique. 

There  are  two  distinct  classes  of  signal.  There  are  signals  in 
time  such  as  speech  or  music;  and  there  are  signals  in  space, 
like  print,  stone  inscription,  punched  ca.  ds,  and  pictures.  Out 
of  all  these  communication  forms,  "’speech"  is  perhaps,  the 
most  "natural"  mode  by  which  human  beings  communicate 
with  each  other.  There  are  also  good  reasons  for  people 
wishing  to  use  speech  to  communicate  with  machines,  it 
must,  however,  be  pointed  out  that  there  is  not  much  empiri¬ 
cal  evidence  to  show  the  value  of  speech  over  other  modes 
of  communication. 

In  a  recent  study  carried  out  by  the  author  [2]  it  was  estab¬ 
lished  that  in  a  tri-service  strategic  C3I  environment  about 
half  of  the  total  traffic  in  Erlangs  was  for  voice  and  the  rest 
was  approximately  equally  divided  between  data  and  mes¬ 
sage  traffic.  In  an  information  theoretic  sense,  however,  the 
bulk  of  communication  was  carried  by  the  message  handling 
system.  About  70%  of  the  traffic  was  for  air  operations.  It  is, 
however,  expected  that  these  proportions  will  change  with 
time  in  favour  of  the  data  traffic.  The  traffic  situation  is,  of 
course,  very  different  in  the  civil  network  where,  at  least  in 
the  foreseeable  future,  voice  service  will  continue  to 
predominate  all  others.  It  must  be  stated  however,  that  the 
main  reason  for  the  prepondera,;'\  of  message  traffic  in 
military  networks  today  is  the  requirement  of  "recording" 
information  in  a  secure  and  easily  accessible  way  and  ability 
to  coordinate  and  disseminate  it. 

Notwithstanding  the  above,  an  experiment  carried  out  at 
Johns  Hopkins  University  [3]  showed  that  teams  of  people 
interacting  together  to  solve  problems  solved  them  much 
faster  using  voice  than  any  other  mode  of  communica¬ 
tion.  There  are  other  studies  which  indicate  that  voice 
provides  advantages  over  other  means  of  communication  for 
certain  applications.  There  is  no  doubt  that  the  main  reason 
for  the  preference  of  voice,  at  least  for  certain  applications, 
stems  from  it  being  'natural",  not  requiring  any  special 
training  to  leam,  and  freeing  the  hands  and  eyes  for  other 
tasks. 

The  features  of  speech  communications  that  are  disad¬ 
vantageous  relate  to  the  difficulty  of  keeping  permanent 
secure  records,  interference  caused  by  competing  environ¬ 
mental  acoustic  noise,  physical  psychological  changes  in  the 
speaker  causing  changes  in  speech  characteristics  or 
disabilities  of  speaking/hearing  and  finallv  its  serial  and 
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informal  nature  leading  to  slower  information  transfer  and  or 
information  access.  It  must  be  pointed  out  however  that  some 
of  the  disadvantages  of  speech  communication  are  depend¬ 
ent  on  the  state  of  technology  and  can  therefore  change  with 
time  and  application. 

Fig.l  shows  how  the  importance  of  the  communication  node 
changes  with  the  phases  of  an  engineering  project  [4).  The 
importance  of  text  dominates  at  the  beginning  and  end  of  an 
engineering  development  process.  In  the  middle  of  the 
process,  other  forms  of  communication  modes  rise  and  fall 
in  importance,  due  to  the  specialised  design  and  implemen¬ 
tation  methods  of  engineering.  Graphics  maintains  its  impor¬ 
tance  throughout  the  process. 

From  the  example  above  it  is  not  too  difficult  to  see  a  certain 
degree  of  resemblance  between  the  modes  of  communication 
required  for  an  engineering  development  project  and  those 
for  command  and  control;  all  modes  are  required  in  general 
with  preference  given  to  some  depending  on  application 
and  the  de%  jlopment  of  technologies  and  operational  con¬ 
cepts. 


2.  COMMUNICATIONS  NETWORKS 

Because  of  the  preponderance  of  traffic  related  to  Air  Opera¬ 
tions  in  militarv  networks  we  shall  now  take  a  brief  look  at 
the  type  of  communications  these  operations  require  and  the 
type  of  environment  in  which  they  are  to  work.  Air  operations 
involve  both  fixed  and  mobile  platforms  (land,  sea  and  air) 
and  communications  that  are  required  to  interconnect  them 
consist  of: 

-  A  switched  terrestrial  network 

-  Air/ground  communications  and 

-  intra-aircraft  (cockpit)  communications. 

These  communications  are  used  to  support: 

-  the  management  of  offensive  air  operations 

-  the  management  of  defensive  aircraft 

-  regional,  sub-regional  air  defence  control  systems. 

in  addition  there  are  also  dedicated  communications 
employed  for  sustained  surveillance,  navigation  and  IFF. 

The  main  air  warfare  missions  and  associated  ranges  together 
with  the  types  of  communications  required  are  given  in  Fig.2. 
These  communications  are  currently  provided  in  NATO  by 
a  combination  of  international  and  national  networks  using 
both  terrestrial  and  satellite  links  together  with  VHF/UHF 
ground/air,  air/air  and  HF  radio  communications  to  and 
between  tactical  strategic  aircraft  (Fig.  3). 

The  terrestrial  transmission  systems  used  today  provide 
nominally  4  kHz  analogue  circuits  even  though  the  NATO 
SATCOM  systems  is  totally  digitised  and  some  national 
systems  (PTT  and  military)  use  digital  transmission  links. 
NATO  also  owns  and  operates  automatically  switched  voice 
and  telegraph  networks.  It  is  to  be  noted  that  a  significant 
portion  of  the  traffic  that  flows  in  the  common-user  network 
is  related  to  air  operations.  As  far  as  UHF/VHF  and  HF 
radios  are  concerned,  they  provide  analogue  voice  and  data 
except  for  JTIDS  (The  Joint  Tactical  Information  Distribu¬ 
tion  System)  which  is  totally  digital  and  is  currently  available 


for  the  NATO  AEW  program.  The  NATO  communications 
systems  carry  so;me  circuits  which  are  cryptographically 
secure  end-to-end  and  there  are  some  links  and  circuits 
carried  by  SATCOM  and  JTIDS  which  are  protected  also 
against  jamming. 

2.1  Integrated  Services  Digital  Network 
(ISDN) 

NATO  decided  in  1984  that  most  of  their  terrestrial  com¬ 
munications  requirements  would  he  met  in  the  future  by 
the  strategic  military  communications  networks ,  that  are 
today  being  designed  and  some  being  implemented  by  the 
Member  Countries.  All  these  networks  largely  follow  the 
CCITT  IDN/1SDN  standards  and  recommendations  and 
adopt  the  International  Standards  Organisation's  (ISO)  Open 
System  Interconnection  Reference  Model  (OSI/RM).  These 
digital  common-user  grid  networks  provide  mission  related, 
situation  oriented,  low-delay  "teleservices"  such  as  plain, 
secure  voice,  facsimile  and  non-interactive  and  interactive 
data  communications.  These  are  enhanced  by  "supplemen¬ 
tary  services"  such  as  priority  and  pre-emption,  secure/non- 
secure  line  warning  as  well  as  closed-user  groups,  call  for¬ 
warding  and  others.  The  switching  subsystem  supports  three 
types  of  connection  methodology  namely,  semi-switched 
connections,  circuit-switched  connections,  and  packet/mes¬ 
sage  switched  connections.  The  circuit  switching  technique 
used  is  the  byte-oriented,  synchronous,  time-division-multi¬ 
plexed  (TDM)  switching  in  accordance  with  CCITT  stand¬ 
ards.  The  basic  channels  are  connected  through  the  network 
as  transparent  and  isochronous  circuit  of  64  kb/s  ornx64  kb/s 
where  n  is  typically  32. 

Future  evolution  of  the  ISDN  will  likely  include  the 
switching  of  broadband  services  at  bit  rates  greater  than  64 
kb/s  ,  at  the  primary  rate,  as  well  as  switching  at  bit  rates 
lower  than  64  kb/s  which  are  made  possible  by  the  end-to-end 
digital  connectivity.  Table  1  shows  some  typical  service 
requirements  for  civil  and  also  for  military  applications. 


Table  1:  Some  Service  Requirements 


Service 

Bandwidth 

Requirements 

ISDN 

Channel 

Type  Circuit  Packet  Channel 

B  D  Switched  Switched  Switched /Overlay 

"Telephone 

8,16,32,64kb/s 

X  X 

*  Interactive  Data 

Communications 

4.8-64  kb/s 

X 

X  X 

"Electronic  Mail 

4.8-64  kbis 

X 

X 

"Bulk  Data  Trans. 

4.8-64  kb/s 

X 

X 

*  Facsimile/ 

4.8-M  kb/s 

X 

X 

Graphics 

"Slow  Scan/ 

56-64  kb/s 

X 

X 

Freeze  Frame  TV 

"Compressed  Video  1.5-2  Mb/s 
Conference  (Primary  rate) 

X  X 

3.  OPERATIONAL  REQUIREMENTS 
3.1  General 

The  requirements  for  air  operations  are  subsumed  in  the  total 
requirement  for  the  switched  networks.  The  network  must  be 


18-3 


dimensioned  to  meet  the  needs  of  non-mobile  military  traffic 
securely,  reliably,  survivably,  and  with  no  operationally  sig¬ 
nificant  delays,  so  as  to  preserve  the  radio-frequency 
spectrum  for  mobile  and  broadcast  applications,  including 
possibly  the  restoration  or  reconfiguration  of  the  static  net¬ 
work  and/or  rerouting  of  traffic  following  battle  damage.  The 
survivability  of  the  communications  must  (at  least)  match 
that  of  the  war  headquarters  and  weapon  sites  which  it 
integrates  and  serves  Operational  procedures  must  be 
developed  to  maintain  essential  operation,  even  when  the 
capacity  of  this  network  has  been  seriously  reduced  by 
battle  damage.  Survivability  of  connectivity  is  however  the 
paramount  importance. 

The  satellite  network  must  similarly  be  dimensioned  to  meet 
the  joint  requirements  of  its  total  user  community  which 
comprises  primarily  those  difficult  to  access  otherwise  be¬ 
cause  of: 

a)  Long  range  (and  relatively  large  data-rate)  require¬ 
ments 

b)  mobility, 

c)  multi-access  requirements. 

Its  security  and  ECM  resistance  must  be  assured  and  its 
potential  any-to-any  and  any-to-all  capability  must  be  made 
available  for  flexible  exploitation  by  the  user. 

Security  and  ECM  resistance  are  equally  required  for  the 
various  tail  links. 

Air-ground  and  ground-air  links  for  close  fighter  control 
cannot  tolerate  delays  of  more  than  a  fraction  of  a  second 
when  they  are  part  of  a  close-control  loop.  The  true  data- 
rate  in  information-theory  terms,  is  not  more  than  100  bits. 
It  is  essential  that  the  interface  to  the  pilot  will  be  user-friend¬ 
ly,  and  this  should  normally  include  (possibly  synthesized) 
spoken  messages,  in  order  to  keep  the  pilot’s  eyes  free  for  his 
primary  duties.  Immunity  to  even  short-term  disruption  by 
ECM  is  essential.  The  air-ground  capacity  required  is  mar¬ 
ginally  smaller  than  that  for  ground-air. 

Broadcast  Control  can  accept  slightly  longer  delays,  but  it 
involves  a  more  varied  type  of  data  and  may  involve  a 
somewhat  larger  total  data  rate;  it  may  also  require  more 
air-ground  traffic.  The  need  for  communications  with  close- 
support  strike  aircraft,  in  a  confused  and  rapidly  changing 
battle  situation  are  similar  to  those  for  fighters,  but  with 
increased  flexibility  and  capacity  in  the  air-ground  direction. 

Long-range  deep-penetration  missions  must  be  accessible  to 
relatively  few  and  short  re-targeting  and  recall  messages.  In 
principle.the  data  rates  need  to  exceed  a  few  bits  («r  second, 
and  delays  of  possibly  several  minutes  could  be  tolerated  if 
necessary.  In  the  reply  direction  acknowledgements  and 
reports  of  survival  or  otherwise,  and  of  success  or  failure  of 
a  strike  mission  are  equally  undemanding  in  terms  of  com¬ 
munications  capacity.  Any  reconnaissance  reports  from  long 
range  could  also  tolerate  a  delay  of  a  few  minutes  if  neces¬ 
sary,  but  even  with  data  reduction,  reconnaissance  reports 
(from  any  range)  can  benefit  from  the  widest  bandwidth 
which  can  be  provided  with  the  technology  available.  For 
long-range  missions,  low  probability  of  intercept  would  also 
be  highly  desirable. 

If  the  technology  dictates  a  sharp  division  in  capability  and/or 


solution  between  operations: 

a)  within  line-of-sight  from  the  ground  behind  the 
FEBA, 

b)  within  line-of-sight  from  the  air  behind  the  FEBA, 

c)  beyond  line-of-sight  from  the  FEBA, 

good,  but  distinct,  solutions  to  these  three  scenarios  can  be 
accepted. 

The  operational  requirements  outlined  above  do  certainly 
imply,  in  addition  to  graphic  and  data  communications, 
the  use  of  voice.  Intelligibility  is  the  most  important 
parameter  with  "speaker  recognition"  aiding  "authentica¬ 
tion"  heing  also  required  although  its  value  in  a  multina¬ 
tional  environment  may  be  questioned. 

3.2  Cockpit  Engineering  [5] 

The  basic  piloting  functions  are  the  following: 

-  flying  (control  of  aircraft  manoeuvres) 

-  navigation  (location  and  guidance) 

-  communications  (voice  and  data  link) 

-  utilities  management 

-  mission  management 

Decisions  to  be  made  by  the  pilot  related  to  the  above  tasks 
are  crucially  influenced  by  how  information  is  obtained, 
and  displayed  and  how  communications  are  processed  and 
handled.  There  is  also  the  problem  of  language  between 
the  machine  and  the  man.  Even  when  the  machine  is  as 
learned  as  the  pilot,  it  will  not  always  know  what  part  of  its 
knowledge  is  to  be  transmitted  to  the  pilot  or  how  to  optimal¬ 
ly  transmit  it. 

It  is  generally  accepted  that  in  many  current  military  aircraft, 
particularly  single-crew  aircraft,  pilot  workload  is  excessive 
and  can  be  a  limit  to  the  capability  of  the  aircraft  as  an 
operational  weapon  system.  Advances  in  on-board  avionics 
systems  have  the  potential  for  generating  more  information, 
and  considerable  care  will  be  required  in  optimising  the 
man-system  interface  in  order  that  the  human  pilot  capability 
(which  will  be  essentially  unchanged)  is  not  a  major  con¬ 
straint  on  overall  system  performance. 

3  .3  The  Use  of  Voice  Systems  in  the  Cockpit 

Visual  signals  are  spatially  concerned;  one  needs  to 
direct  the  field-of-view,  moreover,  in  high  workload  phases 
of  the  mission,  attention  can  be  focused  on  some  types  of 
cisual  information  such  that  other  information  which  sudden¬ 
ly  becomes  important  can  be  "overloaded".  Also  the  amount 
of  information  may  saturate  the  visual  channel  capacity. 
Aural  signals  have  the  advantage  of  being  absorbed  inde¬ 
pendently  of  visual  engagement,  while  man’s  information 
acquisition  capacity  is  increased  by  using  the  two  channels 
simultaneously.  Motoric  skills  are  hardly  affected  by  speak¬ 
ing.  For  information  being  sent  from  aircraft  to  other  humans 
on  the  ground  and  in  the  air,  speech  is  a  natural  and  efficient 
technique  which  has  been  used  for  many  years.  Until  now  the 
process  of  aural  communication  between  aircrew  and  sys¬ 
tems  has  been  usually  restricted  to  a  limited  range  of  warning 
signals  generated  by  the  systems. 
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Digital  voice  synthesis  devices  are  now  widely  available  and 
have  many  commercial  applications.  Technically  there  ap¬ 
pear  to  be  no  problems  in  using  them  in  aircraft  to  transfer 
data  from  aircraft  systems  to  aircrew,  the  real  difficulty 
being  in  identifying  the  types  of  message  which  are  best 
suited  to  this  technique.  Warning  messages  currently  appear 
to  be  a  particularly  useful  application,  though  these  will 
probably  need  to  be  reinforced  by  visual  warnings  as  aircrew 
can  totally  miss  aural  warnings  under  some  conditions.  Feed¬ 
back  of  simple  numerical  data  is  also  being  considered. 

One  of  the  main  disadvantages  of  aural  signals  is  that 
the  intelligibility  is  greatly  impaired  by  noise  in  the  cockpit; 
this  is  true  both  ways.  However,  the  understanding  of  the 
mechanisms  of  speech  synthesis  and  speech  recognition  has 
reached  the  point  where  voice  systems  in  the  cockpit  can  be 
considered.  Although  electronic  voice  recognition  in  the 
laboratory  reaches  scores  of  96  to  98%  (comparable  with 
keyboard  inputs)  the  vocabulary:  is  still  very  limited  and 
recognition  tends  to  be  personalised.  But  the  prospect  of 
logic  manipulation  in  A1  techniques  can  greatly  improve  the 
situation  to  depersonalise  recognition  in  noisy  environments. 
Actual  data  on  such  improvements  are  difficult  to  contain. 
These  would  also  depend  on  how  much  redundancy  is  used 
in  both  syntax  and  semantics.Furthermore  a  coding  'lan¬ 
guage'  is  to  be  preferred  just  as  in  conventional  aircraft  radio 
communication,  to  prevent  the  system  responding  to  un- 
voluntarily  uttered  (emotional)  exclamations. 

Several  commercial  voice  recognition  equipments  are  cur¬ 
rently  available  on  the  open  market,  but  many  of  these  have 
not  been  desig/ned  for  airborne  application  and  considerable 
development  will  be  needed  before  they  can  be  regarded 
as  usable  equipments  for  combat  aircraft.  Simulator  and 
airborne  trials  in  a  number  af  countries  using  this  early 
equipment  have  identified  the  following  as  key  areas  in 
which  further  investigation/improvement  is  required: 

a)  Size  of  vocabulary.  At  present  this  is  very  limited, 
but  recognition  performance  is  generally  inversely 
related  to  vocabulary  size. 

b)  Background  noise  distortion.  The  cockpit  environ¬ 
ment  is  frequently  very  poor,  and  the  oxygen  mask  and 
microphone  are  far  from  ideal. 

c)  Necessity  for  pre-loading  voice  signatures.  Current 
systems  have  to  be  loaded  with  individual  voice 
templates.  Consequently,  if  aircrew  voice  changes 
(e.g..  under  stress)  recognition  performance  is  reduced. 
Moreover  some  subjects  have  a  much  greater  natural 
variability  in  their  voices  than  others. 

d)  Continuous  speech  recognition.  Most  early  equip¬ 
ments  can  only  recognise  isolated  words,  whereas  in 
natural  speech  the  speaker  frequently  allows  one  word 
to  flow  continuously  into  the  next. 

e)  Recognition  Performance.  Even  under  ideal  condi¬ 
tions,  recognition  scores  are  always  less  than  100%  and 
under  bad  conditions  and  with  poor  objects  the 
scores  may  be  only  50-75%c.  Thus  it  is  currently 
necessary  and  probably  will  continue  to  be  so-to  have 
some  form  of  feedback  to  confirm  to  the  speaker  that 


the  message  has  been  correctly  captured. 

In  ummary,  trials  with  first  generation  voice  recognition 
equipments  bave  produced  encouraging  results,  but  the  need 
for  significant  improvements  has  been  identified  and  these 
are  now  being  explored.  It  may  be  too  early  to  give  an  exact 
estimate  of  the  extent  to  which  voice  recognition  techniques 
will  be  used  in  future  combat  aircraft,  but  there  is  consider¬ 
able  promise  that  a  valuable  new  interface  channel  can  be 
developed.  First  applications  are  likely  to  be  in  areas  where 
100%  accuracy  in  data  transmission  is  not  essential  and 
where  an  alternative  form  of  data  input  is  also  available  to 
aircrew. 

4.  SPEECH  PROCESSING 

4.1  General 

Having  established  the  fact  that  the  spoken  word  plays  and 
will  continue  to  play  a  significant  role  in  man  man,  man- 
machine  and  machine-man  communications  for  civil  and 
military  applications  both  real-time  and  with  intermediate 
storage  (e.g.,  for  'voice  mail")  a  brief  look  will  now  be  taken 
at  the  developments  in  speech  processing  that  contribute 
significantly  in  all  these  areas. 

Fig  4  shows  roughly  the  relationship  between  speech  trans¬ 
mission  and  recognition  and  synthesis  of  speech  [6] .  In  each 
case  processing  starts  with  "preprocessing"  which  extracts 
some  important  signal  characteristics.  The  following 
stage  which  is  still  a  preprocessing  stage  but  extracts 
more  complicated  and  combinatorial  parameters  such  as 
segmented  phoneme  parameters  or  prosodic  parameters  like 
speech  intonation  which  are  necessary  for  a  speech  recogni¬ 
tion  system.  The  succeeding  stages  are  concerned  with  the 
central  issue  of  recognition  and  understanding.  A  speech 
output  is  then  produced  based  on  linguistic  rules.  The 
phonetic  and  speech  synthesis  parts  again  handle  the  higher 
and  lower  level  parameters  to  produce  a  speech  signal  which, 
when  applied  to  a  loudspeaker/earpiece,  is  converted  into  an 
acoustic  signal.  In  a  speech  transmission  system  with 
redundancy  reduction  (compression),  the  inner  part  of  Fig  4 
is  by-passed  and  a  parametric  description  of  the  analyzed 
signal  is  directly  sent  to  a  synthesizer  which  can  reproduce 
the  speech  signal. 

4.2  Speech  Coding 

Speech  compression  systems  can  generally  be  classified  as 
either  Waveform  Coders  or  Vocoders,  (i.e.,  voice  coders 
or  analysis-synthesis  telephony).These  two  classes  cover  the 
whole  range  of  compressibility  from  64000  down  to  a  few 
hundred  bits  per  second.  The  important  factors  which  need 
to  be  taken  into  account  when  comparing  different  encoding 
techniques  are  the  speech  quality  achievable  in  the  presence 
of  both  transmission  errors  and  acoustic  noise,  the  data  rate 
required  for  transmission,  the  delay  introduced  by  process¬ 
ing,  the  physical  size  of  the  equipment  and  the  cost  of 
implementation  (a  function  of  coder  complexity  which  can 
be  measured  by  the  oumber  of  multiply-add  operations  re¬ 
quired  to  code  speech,  usually  expressed  in  millions  of 
instructions  per  second  "MIPS"). 

The  most  basic  type  of  waveform  coding  is  pulse  code 
modulation  (PCM)  consisting  of  sampling  (usually  at  8  kHz), 
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quantising  to  a  finite  number  of  levels,  and  binary  encoding. 

There  are  many  variations  on  the  basic  PCM  idea,  the  most 
common  being  differential  encoding  and  adaptive  quantisa¬ 
tion.  Each  variation  has  the  object  of  recuding  the  data  rate 
required  for  a  given  speech  quality,  a  saving  of  approximate¬ 
ly  1  bit  per  sample  (kbit/s)  being  achieved  when  each  is 
optimally  employed.  In  differential  PCM  (DPCM)  the 
sampled  speech  signal  is  compared  with  a  locally  decoded 
version  of  the  previous  sample  sample  prior  to  quantisation 
so  that  the  transmitted  signal  is  the  quantised  difference 
between  samples.  In  adaptive  PCM  (APCM)  the  quantiser 
gain  is  adjusted  to  the  prevailing  signal  amplitude,  either  on 
a  short  term  basis  or  syllabi  cally.  By  controlling  the  adaption 
logic  from  the  quantiser  output,  the  quantiser  gain  can  be 
recovered  at  the  receiver  without  the  need  for  additional 
information  to  be  transmitted.  Adaptive  differential  PCM 
(ADPCM)  is  a  combination  of  DPCM  and  APCM  which 
saves  2  to  4  bits  per  sample  compared  with  PCM,  thus  giving 
48  to  32  kb/s  high  quality  speech. 

There  is  another  adaptive  approach  to  producing  high  quality 
and  lower  bit-rate  coder  which  is  called  'adaptive  subband 
coding'  which  divides  the  speech  band  into  four  or  more 
contiguous  bands  by  a  bank  of  filters  and  codes  each  band 
using  APCM.  After  lowering  the  sampling  rates  in  each  band, 
an  overall  bit  rate  can  be  obtained  while  maintaining  speech 
quality;  by  reducing  the  bits/sample  in  less  perceptually 
important  high-frequency  bands.  Bands  with  low  energy  use 
small  step  sizes,  producing  less  quantisation  noise  that  with 
less  flexible  systems.  Furthermore,  noise  from  one  band  does 
not  affect  other  frequency  bands.  Coders  operating  at  16  kb/s 
using  this  technique  have  been  shown  to  give  high  quality 
but  with  high  complexity  [7). 

When  the  number  of  quantization  levels  in  DPCM  is  reduced 
to  two,  delta  modulation  (DM)  results.  The  sampling  fre¬ 
quency  in  this  case  is  equal  to  the  data  rate,  but  it  has  to  be 
above  the  nyquist  frequency  to  ensure  that  the  binary  quan¬ 
tization  of  the  difference  signal  does  noUproduce  excessive 
quantisation  noise.Just  as  with  PCM,  there  are  many  varia¬ 
tions  of  DM.  The  most  important  from  of  DM  used  in  digital 
speech  communications  is  syllabically  companded  DM; 
there  are  a  number  of  closely  related  versions  of  this,  ex¬ 
amples  being  continuously  variable  slope  DM  (CVSD)  and 
digitally  controlled  DM  (DCDM).  The  data  rate  requirements 
are  a  minimum  of  about  16  kb/s  for  military  tactical  quality 
speech  and  about  48  kb/s  for  commercial  quality. 

When  operated  at  data  rates  of  1 2  kbit/s  and  lower,  the  speech 
quality  obtained  with  PCM  and  DM  coders  is  poor,  and 
consequently  they  cannot  be  used  as  narrow  band  devices. 

Analysis-synthesis  telephony  techniques  are  based  on  a 
model  of  speech  production.  Briefly,  these  are  vocal  tract 
running  from  the  vocal  chords  at  the  top  of  the  larynx  to  the 
mouth  opening  at  the  lips,  and  the  nasal  tract  branching  off 
the  vocal  tract  at  the  velum  and  running  to  the  nose  opening 
at  the  nostrils.  The  glottis  (the  space  between  the  vocal 
chords)  and  the  sub-glottal  air  pressure  from  the  lungs 
together  regulate  the  flow  of  air  into  the  vocal  tract,  and  the 
velum  regulates  the  degree  of  coupling  between  the  vocal  and 
nasal  tracts  (i.e.,  the  nasalisation). 


There  are  two  basic  types  of  speech  sound  which  can  be 
produced,  namely  voiced  and  unvoiced  sounds.  Voiced 
sounds  occur  when  the  vocal  chords  are  tightened  in  such  a 
way  that  the  subglottal  air  pressure  forces  them  to  open  and 
close  quasi-periodically,  thereby  generating  "puffs'of  air 
which  acoustically  excite  the  vocal  cavities.  The  pitch  of 
voiced  sounds  is  simply  the  frequency  at  which  the  vocal 
chords  vibrate.  On  the  other  hand,  unvoiced  sounds  are 
produced  by  forced  air  turbulence  at  a  poind  of  constriction 
in  the  vocal  tract,  giving  rise  to  a  noise-like  excitation,  or 
"hiss". 

In  channel  vocoding  the  speech  is  analyzed  by  processing 
through  a  bank  of  parallel  band-pass  filters,  and  the  speech 
amplitude  in  each  frequency  band  is  digitized  using  PCM 
techniques.  For  synthesis,  the  vocal  and  nasal  tracts  are 
represented  by  a  set  of  controlled  gain,  lossy  resonators,  and 
either  pulses  or  white  noise  are  used  to  excite  them.  In 
pitch-excited  vocoders,  the  excitation  is  explicitly  derived  in 
the  analysis,  whereas  in  voice-excited  vocoders  it  is  derived 
by  non-linear  processing  of  the  speech  signal  in  a  few  of  the 
low  frequency  channels  combined  into  one.  Pitch-excited 
vocoders  require  data  rates  in  the  range  trom  1200  bit/s  to 
2400  bit/s  and  yield  poor  quality  speech,  whereas  voice-ex- 
cited  vocoders  will  provide  reasonable  speech  quality  at  4800 
bit/s  and  good  quality  at  9600  bit/s. 

A  formant  vocoder  is  similar  to  a  channel  vocoder,  but  has 
the  fixed  filters  replaced  by  formant  tracking  filters.  The 
centre  frequencies  of  these  filters  along  with  the  cor¬ 
responding  speech  formant  amplitudes  are  the  transmitted 
parameters.  The  main  problems  in  acquiring  and  maintaining 
lock  on  the  relevant  spectral  peaks  during  vowel-consonant- 
vowel  transitions,  and  also  during  periods  where  the  formants 
become  ill-defined.  The  data  rate  required  for  formant 
vocoders  can  be  as  low  as  600  bit/s,  but  the  speech  quality  is 
poor.  The  minimum  data  rate  required  to  achieve  good 
quality  speech  is  about  1200  bit/s,  but  to  date  this  result  has 
only  been  obtained  using  semi-automated  analysis  with 
manually  interpolated  and  corrected  formant  tracks. 

The  third  method  of  analysis-synthesis  telephony  to  have 
achieved  importance  is  linear  predictive  coding(LPC).  In  this 
technique  the  parameters  of  a  linearised  speech  production 
model  are  estimated  using  mean-square  error  minimisa¬ 
tion  procedures.  The  parameters  estimated  are  not  acoustic 
ones  as  in  channel  and  formant  vocoders,  but  articulatory 
ones  related  to  the  shape  of  the  vocal  tract.  For  a  given  speech 
quality,  a  transmission  data  rate  reduction  in  comparison  with 
acoustic  parameter  vocoding  should  be  achieved  because  of 
the  lower  redundancy  present.  Just  as  with  channel  and 
formant  vocoders,  excitation  for  the  synthesizer  has  to  be 
derived  from  a  separate  analysis,  the  usual  terminology  being 
pitch-excited  or  residual  excited,  corresponding  to  pitch  or 
voice  excitation  in  a  channel  vocoder.  LPC  is  a  very  active 
area  of  speech  research,  and  new  results  appear  regularly.  At 
present  data  rates  as  low  as  2400  bit/s  have  been  achieved  for 
pitch-excited  LPC  with  reasonable  quality  speech,  and  in  the 
range  from  8  kbit/s  to  16  kbit/s  for  residual  excited  LPC  with 
good  speech  quality. 

The  application  of  vector  quantisation  (VQ),  a  fairly  new 
direction  in  source  coding,  has  allowed  LPC  rates  to  be 
dramatically  reduced  to  800  b/s  with  very  slight  reduction  in 
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quality,  and  further  compressed  to  rates  as  low  as  ISO  b/s 
while  retaining  intelligibility  (8,9] .  This  technique  consists 
of  coding  each  set  or  vector  of  the  LPC  parameters  as  group 
instead  of  individually  as  in  scalar  quantisation.  Vector  quan¬ 
tisation  can  be  used  also  for  waveform  coding. 

A  good  candidate  for  coding  at  8  kb!s  is  multipulse  linear 
predictive  coding,  in  which  a  suitable  number  of  pulses  are 
supplied  as  the  excitation  sequence  for  a  speech  segment- 
perhaps  10  pulses  for  a  10-ms  segment.  The  amplitudes  and 
locations  of  the  pulses  are  optimised,  pulse  by  pulse,  in  a 
closed-loop  search.  The  bit  rate  reserved  for  the  excitation 
information  is  more  than  half  the  total  bit  rate  of  8  kb/s.  This 
does  not  leave  much  for  the  linear  predictive  filter  informa¬ 
tion,  but  with  VQ  the  coding  of  the  predictive  parameters 
can  be  made  accurate  enough. 

For  4  kb/s  coding,  code  excited  or  stochastically  excited 
linear  predictive  coding  is  promising.  The  coder  stores  a 
repertory  of  candidate  excitations,  each  a  stochastic,  or  ran¬ 
dom  sequence  of  pulses.  The  best  sequence  is  selected  by  a 
closedloop  search.  Vector  quantization  in  the  linear  predic¬ 
tive  filter  is  almost  a  necessity  here  to  guarantee  that  enough 
bits  are  available  for  the  excitation  and  prediction 
parameters.  Vector  quantization  ensures  good  quality  by 
allowing  enough  candidates  in  the  excitation  and  filter 
codebooks. 

Table  II  below  compares  tradeoffs  for  representative  types 
of  speech  coding  algorithms  [10].It  shows  the  best  overall 
match  between  complexity,  bitrate  and  quality.  A  coder  type 
is  not  necessarily  limited  to  the  bit  rate  stated.  For  example, 
the  medium-complexity  adaptive  differential  pulse-code 
modulation  coder  can  be  redesigned  to  give  communication- 
quality  speech  at  16  kb/s  instead  of  high-quality  speech  at  32 
kb/s.  In  fact,  a  highly  complex  version  can  provide  high- 
quality  speech  at  the  lower  bit  rate.  Similarly  lower-com¬ 
plexity  multipulse  linear  predictive  coding  can  yield  high- 
quality  cording  at  16  kb/s,  and  a  lower-complexity  stochas¬ 
tically  excited  linear  predictive  coder  (LPC)  can  be 
designed  if  the  bit  rate  can  be  8  kb/s  instead  of  4  kb/s. 


Table  II:  Comparison  Low  Bit-Rate  Speech  Coding 
Schemes 


Coder  type 

Bit 

rate 

kb/s 

Complexity  Delay 

MIPS  ms  Quality  MOS 

‘Pulse -code  modulation 

64 

0.01 

0 

High 

‘Adaptive  differential 

pulse-code  modulation 

32 

0.1 

0 

High  >4 

‘Adaptive  subband  coding  16 

1 

25 

High 

‘Multipulse  linear 

predictive  coding 
‘Sloshastically  excited 

8 

10 

35 

Communication 

>2 

linear  predictive  coding 

4 

100 

35 

Communication 

*LPC  vocoder 

7 

1 

35 

Synthetic  <2 

Cost  is  also  a  tradeoff  factor,  but  it  is  hard  to  quantify  in  a 
table.  The  cost  of  coding  hardware  generally  increases  with 
complexity.  However,  advances  in  signal  processor  technol¬ 
ogy  tend  to  decrease  cost  for  a  given  level  of  complexity  and, 
more  significantly,  to  reduce  the  cost  difference  between 
low-complexity  and  high -complexity  techniques. 

Of  course,  as  encoding  and  decoding  algorithms  become 
more  complex  they  take  longer  to  perform.  Complex  algo¬ 
rithms  introduce  delays  between  the  time  the  speaker  utters 
a  sound  and  the  time  a  coded  version  of  it  enters  the  trans¬ 
mission  systems.  These  coding  delays  can  be  objectionable 
in  two-way  telephone  conversations,  especially  when  they 
are  added  to  delays  in  the  transmission  network  and 
combined  with  uncancelled  echoes.  Coding  delay  is  not  a 
problem  if  the  coder  is  used  in  only  one  stage  of  coding  and 
decoding,  such  as  in  voice  storage.  If  the  delay  is  objec¬ 
tionable  because  of  uncancelled  echoes  the  addition  of  an 
echo  canceler  to  the  voice  coder  can  eliminate  or  mitigate  the 
problems.  Finally,  coding  delay  is  not  a  concern  if  the  speech 
is  merely  stored  in  digital  form  for  later  delivery. 

Many  explanations  can  be  given  as  to  why  particular  types 
of  speech  coder  do  not  perform  well  at  low  data  rates.  With 
waveform  coders,  it  is  generally  accepted  that  the  main 
reason  is  excessive  quantisation  noise  despite  compand¬ 
ing  and  or  adaptive  logic.  With  analysis-synthesis  techni¬ 
ques,  the  main  reasons  are  over  simplification  of  the  vocal 
tract  model,  leading  to  imprecise  spectral  characterization, 
and  unreliable  pitch  detection  and  voiced-unvoiced- 
silence  decisions  in  the  analyzer  which,  coupled  with  an 
over-simplified  excitation  model  in  the  synthesizer,  lead  to 
imprecise  temporal  characterisation  and  a  lack  of  natural¬ 
ness  in  the  synthetic  speech. 

A  quantitative  description  of  the  current  state  of  telephone 
speech  coding  in  terms  of  standards  activity,  bit  rate,  and 
MOS  is  summarized  in  Fig.  5 .  The  solid  curves  in  this  figure 
refer  to  generic  examples  of  coding  algorithms  outlined  in 
the  text  above  and  the  broken  curve  represents  a  research  goal 
which  is  regarded  as  achievable.  The  solid  dots  refer  to 
coding  algorithms  that  provide  high  quality  at  64,32  and  16 
kb/s. 

In  conclusion  on  speech  coding,  it  should  be  remarked  that 
there  are  two  complementary  trends  that  are  at  work  in  digital 
telecommunications:  speech  coding  developers  are  trying  to 
reduce  the  bit  rate  for  a  given  quality  level  while  developers 
of  modulation  and  demodulation  techniques  are  endeavour¬ 
ing  to  encrease  the  bit  rate  that  a  channel  of  a  given  bandwith 
can  accommodate. 

4-3  Speech  Synthesis 

Speech  synthesis  involves  the  conversion  of  a  command 
sequence  or  input  text  (words  or  sentences)  into  speech 
waveform  using  algorithms  and  previously  coded  speech 
data.  The  text  can  be  entered  by  keyboard,  optical  character 
recognition,  or  from  a  previously  stored  data  base. 
Speech  synthesizer  can  be  characterized  by  the  size  of  the 
speech  units  they  concatenate  to  yield  the  output  speech  as 
well  as  by  the  method  used  to  code,  store  and  synthesize  the 
speech.  Large  speech  units,  such  as  pl\_rascs  and  sentences 
can  give  high-quality  output  speech  (with  large  memory 
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requirements).  Efficient  coding  methods  reduce  memory 
needs,  but  usually  degrade  speech  quality. 

Synthesizers  can  be  divided  into  two  classes:  text-to-speech 
systems  which  constructively  synthesize  speech  from  text 
using  small  speech  units  and  extensive  linguistic  processing, 
and  voice  response  systems  which  reproduce  speech  directly 
from  previously-coded  speech,  primarily  using  signal 
processing  techniques.  Voice  response  systems  are  often 
called  "speech  coders'  and  contain  both  an  analyzer  and  a 
synthesizer. 

Svnthesizers  can  also  be  classified  by  how  they  parametrize 
speech  for  storage  and  synthesis.  High  quality  systems  with 
large  memory  capacities  synthesize  speech  by  recreating 
the  waveform  sample-by-sample  in  the  time  domain.  More 
efficient  (but  lower  quality)  systems  attempt  to  recreate  the 
frequency  spectrum  of  the  original  speech  from  a  parametric 
representation.  A  third  possibility  is  direct  simulation  of  the 
vocal  tract  movements  using  data  derived  from  X-ray 
analysis  of  human  production  of  specified  sound  sequences. 
Due  to  the  difficulty  of  obtaining  accurate  three  dimensional 
vocal  tract  representations  modelling  the  system  with  a 
limited  set  of  parameters,  this  last  method  usually  yields 
lower  quality  speech  and  bas  yet  to  have  commercial  applica¬ 
tion. 

The  simplest  synthesizers  concatenate  stored  words  or 
phrases.  This  method  yields  high-quality  speech  (depending 
on  the  synthesis  method)  but  is  limited  by  the  need  to  store 
in  computer  (read-only)  memory  all  the  phrases  to  be  syn¬ 
thesized  after  they  have  been  spoken  either  io  isolation  or 
in  carrier  sentences.  For  maximum  naturalness  in  the  syn¬ 
thetic  speech,  each  word  or  phrase  must  originally  be 
pronounced  with  timing  and  intonation  appropriate  for  all 
sentences  in  which  it  could  be  used. 

Hybrid  synthesises  concatenate  intermediate-sized  units  of 
stored  speech  such  as  syllables,  demisyllables,  and  diphones, 
using  smoothing  of  special  parameters  at  the  boundaries 
between  units.  To  further  enhance  thejkexibilityof  sto  red- 
speech  synthesis  systems,  ooe  can  allow  control  of  prosody 
(pitch  and  duration  adjustments)  during  the  synthesis 
process.  With  the  decreasing  cost  of  digital  storage,  stored- 
speech  synthesis  techniques  could  provide  low-cost  voice 
output  for  many  applications. 

It  is  clear  that  stored-speech  systems  are  not  flexible  enough 
to  convert  unrestricted  English  (or  whatever  language)  text 
to  speech.  A  text-to-speech  system  that  uses  synthesis-by 
rule  is  needed  for  applications  such  as  accessing 
electronic  mail  by  voice,  a  reading  machine  etc.  The  text-to- 
speech  system  must  convert  incoming  text,  such  as  electronic 
mail,  that  often  includes  abbreviations,  Roman  numerals, 
dates,  times,  formulas,  and  a  wide  variety  of  punctuation 
marks  into  some  reasonable,  standard  form.  The  text  must  be 
further  translated  into  a  broad  phonetic  transcription. 

There  are  several  commercial  text-to-speech  conversion 
systems  in  the  market  which  come  in  board,  peripheral, 
software  or  system  form.  They  are  mostly  for  English  adult 
male  but  some  do  adult  female  and  child  voice.  The  speech 
mode  used  is  mostly  words  with  some  iccepting  also  letters. 


The  synthesis  technique  employed  is  mostly  formant  syn¬ 
thesis  but  some  manufactures  use  LPC.  Prices  vary  from  a 
few  hundred  Dollars  for  software  to  a  few  tens  of  thousand 
Dollars  for  systems.  The  quality  of  even  the  best  systems  is 
such  that  during  tests,  listeners  understood  the  synthetic 
speech  produced  97.7%  of  the  time  compared  with  99.4% 
tior  human  speech.  Research  in  text-to  speech  synthesis 
which  concentrates,  at  present,  on  producing  speech  that 
sounds  more  natural,  is  expected  to  provide  systems  which 
are  more  flexible  for  selecting  the  speaker  characteristics, 
different  languages  and  their  dialects,  and  regional 
variabilities. 

4.4  Speech  Recognition  [14] 

Of  all  the  speech  processing  techniques,  speech  recognition 
is  the  most  intractable  one.  The  ultimate  objective  of  most 
research  in  this  area  is  to  produce  a  machine  which  would 
understand  conversational  speech  with  unrestricted 
vocabulary,  from  essentially  any  talker.  We  are  far  from  this 
goal. 

The  reason  why  automatic  speech  recognition  is  such  a 
difficult  problem  can  be  stated  very  briefly  under  four  prob¬ 
lem  areas:  First,  the  speech  signal  is  normally  continuous  and 
there  are  no  acoustic  markers  which  identify  the  word  boun¬ 
daries.  Second,  speech  signals  are  highly  variable  from  per¬ 
son  to  person  and  even  in  one  and  the  same  person  depending 
on  his  state.  The  third  problem  area  is  ambiguity  which  is 
characterised  by  conditions  whereby  patterns  which  should 
be  different  end  up  looking  alike.  The  fourth  problem  area 
results  from  the  fact  that  the  speech  signal  is  a  part  of  the 
complex  system  of  human  language  where  it  is  often  the 
intention  behind  a  message  that  is  more  important  than  the 
message  itself.  Therefore  an  advanced  speech  recogniser 
would  be  expected  to  incorporate  techniques  which  would 
enable  it  to  use  the  meanings  of  words  in  order  to  interpret 
what  bas  been  said.  However,  there  are  several  applications 
which  do  not  require  this  full  capability.  They  range  from 
voice  editors,  and  information  retrieval  from  data  bases  to 
basic  English  and  large  vocabulary  systems  required  for 
office  dictalion/word  processing  and  language  translation. 

A  technology  that  is  closely  related  to  speech  recognition 
is  speaker  recognition,  or  automatic  recognition  of  a  talker 
from  measurements  of  individual  characteristics  in  the 
voice  signal.  The  two  tasks  that  are  relevant  here  are  "ab¬ 
solute  identification"  and  "talker  verification"  the  former 
being  the  more  difficult  to  perform.  An  interesting  military 
application  of  speaker  recognition  is  related  to  the  monitor¬ 
ing  of  enemy  radio  channels  with  a  view  to  identifying, 
perhaps  in  conjunction  with  keyword  recognition,  critical 
situations  before  they  occur. 

The  recognition  problem  has  at  least  three  dimensions: 
vocabulary  size,  speaker  identity  and  fluency  of  input  speech 
and  the  performance  of  speech  recogniser  also  depend  on 
the  acoustic  environment  and  transmission  conditions. 
Current  understanding  permits  building  practical  systems 
that  reliably  recognise  several  hundred  words  spoken  by  a 
person  who  trained  the  system.  Recognition  for  any  or  all 
speakers  requires  about  ten  times  more  computation  than 
for  individuals  whose  vocabulary  patterns  have  been  stored. 
Recognition  of  single  words  or  short  phrases-spoken  in  isola- 
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tion-can  be  done  reliably,  even  over  dialed-up  telephone 
channels.  Recognition  of  connected  words  is  under  active 
development.  Recognition  of  conversational  fluent  speech  is 
in  fundamental  research,  and  advances  st.  ngly  depend  on 
good  computational  models  for  syntax  and  semantics. 

When  setting  out  to  define  the  problems  associated  with 
implementing  a  speech  recognition  system,  one  finds  that 
there  are  a  number  of  general  issues  that  must  be  resolved 
before  designing  and  building  the  system.  One  such  issue  is 
the  size  and  complexity  of  the  user  vocabulary.  Although 
useful  recognition  systems  have  been  built  with  as  few  as  two 
words  (yes,  no),  there  are  at  least  four  distinct  ranges  of 
vocabulary  size  of  interest.  Very  small  vocabularies  (on  the 
order  of  10  words)  are  most  useful  for  control  tasks  -  e.g.  all 
digit  dialing  of  telephone  numbers,  repertory  name  dialing, 
access  control  etc.  Generally  the  vocabulary  words  are 
chosen  to  be  highly  distinctive  words  (i.e.  of  low  complexity) 
to  minimize  potential  confusions.  The  next  range  of 
vocabulary  size  is  moderate  vocabulary  systems  having  on 
the  order  of  100  words.  Typical  apf  lications  include  spoken 
computer  languages,  voice  editors,  information  retrieval 
from  databases,  controlled  access  via  ■celling  etc.  For  such 
applications,  the  vocabulary  is  general'.,  irly  complex  (i.e. 
not  all  pairs  of  words  are  highly  distinctive),  but  word  con¬ 
fusions  are  often  resolved  by  the  syntax  of  the  specific  task 
to  which  the  recognizer  is  applied.  The  third  vocabulary 
range  of  interest  is  the  large  vocabulary  s  ystem  with 
vocabulary  sizes  on  the  order  of  1000  words.  Vocabulary 
sizes  this  large  are  big  enough  to  specify  fairly  comfortable 
subsets  of  English  and  hence  are  used  for  conversational 
types  of  applications  -  e.g.  the  IBM  laser  patent  text,  basic 
English,  etc.  [12,13].  Such  vocabularies  are  inherently  very 
complex  and  rely  heavily  on  task  syntax  to  resolve  recogni¬ 
tion  ambiguities  between  similar  sounding  words.  Finally 
the  last  range  of  vocabulary  size  is  the  very  large  vocabulary 
system  with  10,000  words  or  more.  Such  large  vocabulary 
sizes  are  required  for  office  dictation/word  processing  and 
language  translation  applications. 

Although  vocabulary  size  and  complexity  is  of  paramount 
importance  in  specifying  a  speech  recognition  system, 
several  other  issues  can  also  greatly  affect  the  performance 
of  a  speech  recognizer.  The  system  designer  must  decide  if 
the  system  is  to  be  speaker  trained,  or  speaker  independent; 
the  format  for  talking  must  be  specified  (e.g.  isolated  inputs, 
connected  inputs,  continuous  discourse);  the  amount  and 
type  of  syntactic  and  semantic  information  must  be  specified 
the  speaking  environment  and  transmission  conditions  must 
be  considered  etc.  The  above  set  of  issues,  by  no  means 
exhaustive,  gives  some  idea  as  to  how  complicated  it  can  be 
to  talk  about  speech  recognition  by  machine. 

There  are  two  general  approaches  to  speech  recognition  by 
machine,  the  statistical  pattern  recognition  approach,  and  the 
acoustic -phonetic  approach.  The  statistical  pattern  recogni¬ 
tion  approach  is  based  on  the  philosophy  that  if  the  system 
has  “seen  the  pattern,  or  something  close  enough  to  it,  before, 
it  can  recognize  it."  Thus,  a  fundamental  element  of  the 
statistical  pattern  recognition  approach  is  pattern  training. 
The  units  being  trained,  be  they  phrases,  words,  or  subword 
units,  are  essentially  irrelevant,  so  long  as  a  good  training  set 
is  available,  and  a  good  pattern  recognition  model  is  applied. 
On  the  other  hand,  the  acoustic-phonetic  approach  to  speech 


recognition  has  the  philosophy  that  speech  sounds  have 
cenain  invariant  (acoustic)  properties,  and  that  if  one  could 
only  discover  these  invariant  properties,  continuous  speech 
could  be  decoded  in  a  sequential  manner  (perhaps  with  delays 
of  several  sounds).  Thus,  the  basic  techniques  of  the 
acoustic-phonetic  approach  to  speech  recognition  are  feature 
analysis  (i.e.  measurement  of  the  invariants  of  sounds),  seg¬ 
mentation  of  the  feature  contours  into  consistent  groups  of 
features,  and  labelling  of  the  segmented  features  so  as  to 
detect  words,  sentences,  etc.  To  date,  the  greatest  success 
in  speech  recognition  have  been  achieved  using  the  pattern 
recognition  approach  [14]. 

Figure  6  shows  a  biock  diagram  of  the  pattern  recognition 
model  used  for  speech  recognition.  The  input  speech  signal, 
s(n),  is  analyzed  (based  on  some  parametric  model)  to  give 
the  test  pattern,  T,  and  then  compared  to  a  prestored  set  of 
reference  patterns,  (  R  v ) ,  1  < v  <  V  (corresponding  to  the  V 
labelled  .patterns  in  the  system)  using  a  pattern  classifier  (i.e. 
a  similarity  procedure).  The  pattern  similarity  scores  are 
then  sent  to  a  decision  algorithm  which,  based  upon  the 
syntax  and/or  semantics  of  the  task,  chooses  the  best 
transcription  of  the  input  speech. 


Results  On  Isolated  Word  Recognition 

Using  the  pattern  recognition  model  of  Fig.  6  with  an  8111  order 
LPC  parametric  representation,  and  using  the  non-parametric 
template  approach  for  reference  patterns,  a  wide  variety  of 
tests  of  the  recognizer  have  been  performed  with  isolated 
word  inputs  in  both  speaker  dependent  (SD)  and  speaker 
independent  (SI)  modes.  Vocabulary  sizes  have  ranged 
from  as  small  as  10  words  (i.e.  the  digits  zero-nine  )  to  as 
many  as  1 109  words. 

Table  111  gives  a  summary  of  recognizer  performance  under 
the  conditions  discussed  above.  It  can  be  seen  that  the  result¬ 
ing  error  rates  are  not  strictly  a  function  of  vocabulary  size, 
but  also  are  dependent  on  vocabulary  complexity.  Thus 
a  simple  vocabulary  of 200  polysyllabic  Japanese  city  names 
had  a  2.7%  error  rate  (in  an  SD  mode),  whereas  a  complex 
vocabulary  of  39  alphadigit  terms  (in  both  SD  and  SI  modes) 
had  error  rates  of  on  the  order  of  4.5  to  7.0%. 

Table  III  also  shows  that  in  cases  where  the  same  vocabulary 
was  used  in  both  SD  and  SI  modes  (e.g.  the  alphadigits  and 
the  airline  words),  the  recognizer  gave  reasonably  com¬ 
parable  performances.  This  result  indicates  that  the  SI  mode 
clustering  analysis,  which  yielded  the  set  of  SI  templates  or 
models,  was  capable  of  providing  the  same  degree  of  repre¬ 
sentation  of  each  vocabulary  word  as  either  casual  or  robust 
training  for  the  SD  mode.  Sometimes  the  computation  of  the 
SI  mode  recognizer  was  higher  than  that  required  for  the  SD 
mode  whenever  a  larger  number  of  templates  were  used  in 
the  pattern  similarity  comparison. 
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Table  ID:  Performance  of  Template-Based 
Isolated  Word  Systems 


Error 

Vocabulary  Mode  Rate  (%) 


10  Digits 

SI 

0.1 

37  Dialer  Words 

SD 

0 

39  Alphadigits 

SD 

4.5 

SI 

7.0 

34  Computer  Terms 

SI 

3.5 

129  Airline  Worts 

SD 

1.0 

SI 

2.9 

200  Japanese  Cities 

SD 

2.7 

1109  Basic  English 

SD 

4.3 

Connected  Word  Recognition  Model 

Typical  performance  results  for  connected  word  recognizers, 
based  on  a  level  building  implementation,  are  shown  in 
Table  IV.  For  a  digits  vocabulary,  string  accuracies  greater 
than  99%  have  been  obtained.  For  name  retrieval,  by  spell¬ 
ing,  from  a  17,000  name  directory,  string  accuracies  of  from 
90%0  to  96%  have  been  obtained.  Finally,  using  a  moderate 
size  vocabulary  of  127  words,  the  accuracy  of  sentences  for 
obtaining  information  about  airlines  schedules  is  between 
90%  and  99%.  Here  the  average  sentence  length  was  close 
to  10  words.  Many  of  the  errors  occurred  in  sentences  with 
long  strings  of  digits. 

Continuous,  Large  Vocabulary, 

Speech  Recognition 

The  area  of  continuous,  large  vocabulary,  speech  recognition 
refers  to  systems  with  at  least  1000  words  in  the  vocabulary, 
a  syntax  approaching  that  of  natural  English  (i.e.  an  average 
braaching  factor  on  the  order  of  100),  and  possibly  a  semantic 
model  based  on  a  given,  well  defined,  task.  For  such  a 
problem,  there  are  three  distinct  sub-problems  that  must  be 
solved,  namely  choice  of  a  basic  recognition  unit  (and  a 
modelling  technique  to  go  with  it),  a  method  of  mapping 
recognized  units  into  words  (or,  more  precisely,  a  method  of 
scoring  words  from  the  recognition  scores  of  individual  word 
units),  and  a  way  of  representing  the  fomial  syntax  of  the 
recognition  task  (or,  more  precisely,  a  way  of  integrating 
the  syntax  directly  into  the  recognition  algorithm). 

For  each  of  the  three  parts  of  the  continuous  speech  recogni¬ 
tion  problem,  there  are  several  alternative  approaches.  For 
the  basic  recognition  unit,  one  could  consider  whole  words, 
half  syllables  such  as  dyads,  demisyl  lables,  or  di phones,  or 
sound  units  as  small  as  phonemes  or  phones.  Whole  word 
units,  which  are  attractive  because  of  our  knowledge  of  how 
to  handle  them  in  connected  environments,  are  today  imprac¬ 
tical  to  train  since  each  word  could  appear  in  a  broad  variety 
of  contexts.  Therefore  the  amount  of  training  required  to 
capture  all  the  types  of  word  environments  is  unrealistic.  For 
the  sub-word  units,  the  required  training  is  extensive,  but 
could  be  carried  out  using  a  variety  of  well  known,  existing 
training  procedures.  A  full  system  would  require  between 
1000  and  2000  half  syllable  speech  units.  For  the  phoneme¬ 
like  units,  only  about  30-100  units  would  have  to  be  trained. 

The  problem  of  representing  vocabulary  words,  in  terms  of 


the  chosen  speech  unit,  has  several  possible  solutions.  One 
could  create  a  network  of  linked  word  unit  models  for  each 
vocabulary  word.  The  network  could  be  either  a  determinis¬ 
tic  (fixed)  or  a  stochastic  structure.  An  alternative  is  to  do 
lexical  access  from  a  dictionary  in  which  all  word  pronun¬ 
ciation  variants  (and  possibly  part  of  speech  information)  are 
stored,  along  with  a  mapping  from  pronunciation  units  to 
speech  representation  units. 

Finally  the  problem  of  representing  the  task  syntax,  and 
integrating  it  into  the  recognizer,  has  several  solutions.  The 
task  syntax,  or  grammar,  can  be  represented  as  a  determinis¬ 
tic  state  diagram,  as  a  stochastic  model  (e.g.  a  model  of  word 
tngram  statistics),  or  as  a  formal  grammar.  There  are  ad¬ 
vantages  and  disadvantages  to  each  of  these  approaches. 

To  illustrate  the  state  of  the  art  in  large  vocabulary  speech 
recognition,  consider  the  results  shown  in  Table  V .  The  table 
shows  results  for  two  tasks,  namely  an  office  dictation 
system  [15]  ,  and  a  naval  resource  management  system 
[16-2 1  ].  The  office  dictation  system  uses  phoneme-like  units 
in  a  statistical  model  to  represent  words,  where  each 
phoneme-like  unit  is  a  statistical  model  based  on  vector- 
quantized  spectral  outputs  of  a  speech  spectrum  analysis.  A 
third  statistical  model  is  used  to  represent  syntax;  thus  the 
recognition  task  is  essentially  a  Bayesian  optimization  over 
a  triply  embedded  sequence  of  statistical  models.  The  com¬ 
putational  requirements  are  very  large,  but  a  system  has  been 
implemented  using  isolated  word  inputs  for  the  task  of  auto¬ 
matic  transcription  of  office  dictation.  For  a  vocabulary  of 
5000  words,  in  a  speaker  trained  mode,  with  20  minutes  of 
training  for  each  talker,  the  average  word  error  rates  for  5 
talkers  are  2%  for  prerecorded  speech,  3. 1  %  for  read  speech, 
and  5.7%  for  spontaneously  spoken  speech  [14) .  The  naval 
resource  management  system  also  uses  a  set  of  about  2000 
phoneme-like  units  (PLU's)  to  represent  words  where  each 
PLU  is  a  statistical  model  of  a  phoneme  in  specified  con- 
texts.The  task  syntax  (that  of  a  ships  database)  is  specified  in 
the  form  of  a  finite  state  network  with  a  word  pair  grammar 
with  average  word  branching  factor  (perplexity)  of  60. 
When  tested  on  a  vocabulary  of  991  words,  in  a  speaker 
independent  mode,  using  continuous,  fluently  spoken,  sen¬ 
tences,  a  word  error  rate  of  about  4.5  %  was  obtained  [20]. 


Summary 

The  challenges  in  speech  recognition  are  many.  As  outlined 
above  the  performance  of  current  systems  is  barely  accept¬ 
able  for  large  vocabulary  systems,  even  with  isolated  word 
inputs  speaker  training,  and  favourable  talking  environ¬ 
ment.  Almost  every  aspect  of  continuous  speech  recogni¬ 
tion,  from  training  to  systems  implementation,  represents  a 
challenge  in  performance,  reliability,  and  robustness. 
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Fig.l.  Encoding  of  Communications  in  an  Engineering 
Development 


Fig.2.  Air  Warfare  Missions  and  Range 
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Fig.3.  Communications  Zones  and  Means 


Fig. 4.  Relation  Between  Different  Speech  Bandwidth 
Compression  and  Coding  Techniques  [6] 


Fig.5.  3.2  kHz  Speech  Quality  and  Transmission  Rate  [1 1] 


/WORDS,  PHRASES, \ 
\  SYLLABLES  / 


Figure  6.  Pattern  Recognition  Model  for  Speech  Recognition. 
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vocabulary 

MODE 

WORD 

ACCURACY 

TASK 

STRING  (TASK) 
ACCURACY 

Digits 
(10  words) 

Speaker  Dependent 
or 

Speaker  Independent 

99.8%  SI 

99.9%  SD 

1-7  Digit 
Strings 

1-7  Digit 
Strings 

99.2%  SI* 

99.6%  SD* 

Letters  of  the 
Alphabet 
(26  words) 

Speaker  Dependent 
or 

Speaker  Independent 

=  90% 

Directory  Listing 
Retrieval 
(17,000  Name 
Directory) 

96%  SD 

90%  SI 

Airline  Terms 
(129  words) 

Speaker  Dependent 
or 

Speaker  Independent 

99.9%  SD 
97%  SI 

Airline 

Information 

and 

Reservations 

99%  SD 

90%  SI 

*  Known  string  length. 


Table  IV 

Performance  of  Connected  Word  Recognizers  on 
Specific  Recognition  Tasks 


[Task 

Syntax 

Mode 

Vocabulary 

Word  Error  Rate 

[Office 

Direction 

(IBM) 

Word  Trigram 
(Perplexity=  100) 

SD. 

Isolated 

Word  Input 

5000  Words 

2%-prerecorded  speech 
3.1%-read  speech 
5.7%-spontaneous  speech 

Naval 

Resource 

Managements 

(DARPA) 

Finite  State 

Grammar 

(Perplexity=60) 

SI. 

Fluent 

Speech  Input 

991  Words 

4.5% 

Table  V 

Performance  of  Large  Vocabulary  Speech  Recognizers 
on  Specific  Recognition  Tasks 
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ABSTRACT 


This  paper  describes  the  challenges  that  lie  in  the 
development  and  design  of  a  pilot  vehicle  interface  (PVI), 
both  in  the  basic  voice  processing  technologies  as  in  the 
robustness  requirements  of  the  system ,  due  to  the  peculiar 
circumstances  in  which  it  has  to  be  used . 

Furthermore ,  we  will  focus  on  the  state  of  the  art,  and  on 
the  results  of  the  current  R&D  efforts  within  Lernout  & 
Hauspie  Speech  Products  on  both  the  recognizer  itself  as 
well  as  its  robustness,  and  also  the  hardware 
implementation . 

At  the  end,  we  will  dip  into  the  future  and  look  at  the 
continuing  R&D  efforts  both  to  enhance  the  available 
algorithms,  and  to  undertake  new  basic  efforts  in  the  area 
of  application. 


1.  INTRODUCTION . 

Amongst  the  most  important 
technologies  for  improving 
man-^machine-interf  ace  (MMI), 
voice  dialogue  occupies  an 
important  position.  The 
recent  progress  in  speech 
processing  has  led  to  high 
quality  synthesizers  and  to 
reliable  real-time 
recognizers . 

The  activities  of  a  pilot  in 
(military)  helicopters  are 
similar  to  those  in  fighter 
aircraft.  Therefore,  there 
exist  opportunities  for 
speech  recognition  and  for 
voice  synthesis  in  hands- 
busy,  eyes-busy,  heavy  work¬ 
load  situations. 


However,  the  specific 
characteristics  of  the 
helicopter  environment,  with 
a  more  difficult  acoustic 
noise  problem,  require  a 
special  kind  of  robustness 
of  the  recognizer.  Almost 
all  of  the  important  special 
circumstances  are  present 
during  a  helicopter  flight, 
with  the  exception  of  G- 
forces  and  oxygen  mask 
noise. 

Moreover,  it  is  generally 
accepted  that  a  helicopter 
is  more  difficult  to  fly 
than  a  winged  aircraft. 

These  reasons  explain  the 
option  to  operate  on  a 
rotor  aircraft  rather  than 
on  a  fixed  wing  aircraft. 
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2. TECHNOLOGICAL  CHALLENGES. 

The  technological  challenges 
that  are  faced  in  the  design 
and  development  of  a  vocal 
MMI  are  situated  on  three 
basic  levels  .  First,  there 
are  the  basic  voice 
processing  technologies  that 
are  currently  emerging  and 
available  in  an  office-type 
environment  :  low  bit  rate 
coding,  text-to-speech  and 
voice  recognition. 

Secondly,  the  application  of 
these  technologies  into  a 
cockpit-environment  faces 
the  researchers  with  the 
robustness  of  their 
algorithms.  The  performance 
of  these  algorithms  is 
influenced  by  all  kinds  of 
external  factors  such  as  the 
Lombard-effect,  the 
background  noise,  the  use  of 
oxygen  mask,  the  pilot 
stress  and  G-forces. 

And  finally,  there  are  the 
hardware  challenges  due  to 
the  trade  off  between  the 
real  time  and  the  processing 
power  requirement.  Indeed 
the  hardware  has  to  be 
fitted  in  into  a  limited 
space,  and  coupled  with  the 
existing  communication  and 
avionics  equipment. 


2.1  Basic  Technologies. 

On  the  level  of  the  basic 
voice  processing 
technologies,  one  will  find 
the  speech  recognizer, 
enabling  the  pilot  to  input 
commands  into  the  system; 
and  the  audio  output  system, 
using  low  bit  rate  coding 
and  synthetic  voice  for 
audio  feedback  purposes,  or 
for  warning  systems. 

2.1.1.  Recognizing  speech 
amounts  to  comparing  an 
utterance  with  some 
references.  The  variability 
of  different  pronunciations 
by  different  speakers  make 
this  matching  difficult  due 


to  the  high  variability  of 
the  input  signals,  in 
particular  time  warping 
effects . 

The  recognition  rate 

typically  is  a  function  of 
the  following  parameters  : 
the  size  of  the  active 
lexicon,  the  dependency  on 
the  speaker  and  the 

operating  mode  i.e. 

isolated  words  or  continuous 
speech.  But  also  other 
elements  affect  the 

performance  of  the 

algorithm,  such  as  the 
preprocessing  and  the  kind 
of  modelling. 

After  a  preprocessing  phase, 
which  essentially  aims  at 
the  reduction  of  the  data 
volume  by  frequency 

analysis,  generally  Hidden 
Markov  Models  or  HMM 
(amongst  the  different  voice 
recognition  algorithms)  are 
used  for  both  isolated  words 
and  for  continuous  speech, 
possibly  in  combination  with 
neural  networks  [ 1 ] [ 3 ] [ 9 ] . 

In  essence,  HMM  are 

parametric  models  of  the 
stochastic  production  of  a 
vocal  signal  and  describe 
voice  units  such  as  words 
and  phonemes. 

Any  utterance  corresponds  to 
a  path  in  the  graph  of  an 
HMM.  The  statistical  nature 
of  this  graph  brings  enough 
flexibility  for  coping  with 
the  time  warping  problem  and 
simultaneously  provides  a 
probabilistic  score  for  the 
decision  process. 

Recent  improvements  of  the 
HMM  are  the  use  of 
durational  constraints  [11- 
13],  of  multi-gaussian  [10] 
and/or  semi-continuous 

density  functions  [2]. 
According  to  the  nature  of 
the  probability 

distributions  involved,  a 
vector  prequantification  of 
the  acoustic  space  could  be 
required  :  K-means  [5]  and 
LVQ  [ 6 ] . 
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By  the  use 
statistical 
techniques 
Maximization 
algorithms) , 


of  standard 
analysis 
( Expectation 
or  Viterbi 
one  can 


determine  the  parameters  of 
the  standard  models  on  the 
basis  of  a  large  number  of 
utterances  of  the  units. 
This  is  called  training. 


A  speaker-dependent  system 
is  more  accurate  than  an 
independent  system, 
generally  used  for  large 
public  applications.  There 
are  several  solutions  to 
achieve  this  dependency, 
without  adding  cumbersome 
procedures  :  dynamic  speaker 
adaptation  (training)  during 
the  pre-flight  check- list, 
or  the  use  of  a  personal 
chip-card,  on  which  pilot 
specific  information  is 
stored,  such  as  his  seat 
position  and  his  voice 
profile. 

The  size  of  the  lexicon  roust 
and  can  functionally  be 
rather  limited.  Experience 
has  shown  that  a  pilot  who 
has  to  M  remember"  too  many 
words  as  commands,  often  has 
to  think  about  the  ' right' 
command  to  use,  which  is  a 
waste  of  time  in  critical 
situations.  The  vocabulary 
can  therefore  be  limited  to 
250  to  300  words.  However, 
the  real  challenge  lies  in 
achieving  a  low  perplexity 
of  the  vocabulary,  that  is 
the  size  of  the  largest 
subset  of  the  lexicon  that 
is  active  at  any  given 
moment  during  the 
recognition. 


2.1.2.  Also  on  the  first 
level ,  one  must  consider  the 
use  of  text-to-speech  for 
warning  messages  and  for 
audio- feedback  to  the 
recognizer.  However,  also 
prerecorded  messages,  using 
a  low  bit  rate  coding 
algorithm,  as  a  voice 


output  technique  could  have 
advantages,  in  particular 
for  warning  systems  with 
short  messages,  although  the 
updating  of  the  messages 
will  prove  to  be  a  complex 
matter.  Text-to-speech  then 
again  will  prove  to  be  more 
appropriate  for  the 

generation  of  long  variable 
sentences  in  variable 
situations. 

2.2  Robustness. 

On  the  second  level ,  the 
PVI -system  has  to  deal  with 
environmental  disturbances 
affecting  in  particular  the 
recognizer. 

2.2.1.  The  most  obvious 
challenge  lies  in  the 
problem  of  background  noise. 
Note  that  a  helicopter  pilot 
does  not  wear  a  face  mask, 
which  would  reduce  acoustic 
noise  in  the  microphone.  In 
general ,  existing  studies 
relate  to  the  processing  of 
the  disturbed  signal  in 
order  to  find  a  form  which 
is  more  agreeable  to  tie 
listener.  Classical  methods 
are  adaptive  filtering  and 
spectral  substraction,  while 
more  recently  methods  based 
on  neural  networks  and 
hidden  Markov  noise  models 
are  being  studied  [7], 
However,  the  purpose  of  the 
research  is  not  to  enhance 
the  auditive  comfort,  but  to 
increase  the  recognition 
rate. 

Next  to  that,  the  noise  in 
the  cockpit  can  exhibit  some 
regularity  (engine  or 
propeller  noise,  pilot's 
breathing  in  an  oxygen 
mask).  These  are  no 
precarious  signals,  and  one 
could  consider  original 
methods  using  correlation 
methods  or  trained  noise 
models . 


2.2.2.  Also  in  a  helicopter, 
there  will  be  a  drop  in 
recognition  rate  due  to  the 
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mechanical  vibrations 
affecting  the  pilot, 
changing  the  conditions  of 
speech  production. 

2.2.3.  A  related  disturbing 
factor  is  the  so-called 
LOMBARD-ef feet  [8],  defined 
as  the  reflex  that  takes 
place  when  the  speaker 
modifies  his  vocal  effort 
while  speaking  in  the 
presence  of  noise.  With 
shouted  speech,  the  vocal 
effort  increases  the  energy, 
but  decreases  the 
intelligibility.  In 
particular,  there  is 
distortion  of  consonants 
when  speech  is  produced  in 
noise.  This  effect  has  to  be 
taken  into  account  at  the 
moment  of  the  training  of 
the  system  :  the  training 
has  to  be  performed  under 
exactly  the  same  (noise) 
circumstances,  or  the  pilot 
has  to  be  trained  against 
the  LOMBARD-ef feet  [8]. 

2.2.4.  Speech  generated 
under  stress  can  cause 
recognition  problems . 
Indeed,  stressed  speech  is 
characterized  by  a  sequence 
of  non-words,  stops  or  lack 
of  any  grammar. 


2.3  Hardware. 

Finally,  an  important 
challenge  lies  in  the  real 
time  versus  the  processing 
power  requirement.  Indeed, 
the  systems  must  be  ported 
on  a  relatively  small 
hardware  platform  with 
enough  memory  and  processing 
power  to  assure  a  prompt  and 
timely  response  of  the 
system  to  the  voice  input, 
without  reducing  the 
performances  of  the  existing 
embarked  computers. 


3.  STATE  OF  THE  ART. 

The  current  research  efforts 


have  been  concentrated  on 
the  following  levels  :  the 
basic  technologies  and  their 
robustness  (3.1)  and  their 
implementation  on  a  hardware 
platform  (3.2). 

3.1  Basic  Technologies. 

3.1.1.  At  the  front  level  of 
a  recognition  system  lays 
the  signal  preprocessing, 
which  includes  anti-aliasing 
filtering  and  sampling.  The 
large  amount  of  produced 
speech  data  is  compressed  in 
the  frequency  domain.  Since 
speech  can  be  considered  as 
a  stationary  process  on 
short  time  intervals  only,  a 
set  of  parameters  is 
computed  per  time  frame  of 
10ms.  LPC  analysis  and 
cepstral  analysis  are  the 
most  widespread  techniques. 
With  both  of  them,  the 
signal  gets  rid  of  the 
pitch,  the  fundamental 
frequency  of  the  vocal 
chords  which  carries 
irrelevant  information  on 
the  speaker  himself,  and  of 
the  prosody. 

More  contextual  information 
(at  the  level  of  the 
acoustic  representation)  is 
retained  by  adding  to  the 
cepstral  coefficients 
information  about  the 
cepstral  differences  between 
neighbouring  frames. 
Similarly,  contextual  input 
in  multilayer  perceptrons 
(neural  nets)  has  also  led 
to  improved  phonetic 
classification  [ 1 ] . 

3.1.2.  The  second  step  is 
the  definition  of  models, 
based  on  a  description  of 
phonemes;  a  word  model  is 
then  built  by  concatenating 
phonemic  models,  which 
enables  us  to  add  new  words 
to  the  lexicon  simply  by 
providing  their  phonemic 
transcriptions . 

An  efficient  description  of 
the  lexicon  has  been 
achieved  by  creating  a 


19-5 


lexical  tree.  Here,  each 
word  is  described  by  a  path 
starting  at  the  root  and 
ending  in  a  node  or  a  leaf 
of  the  tree.  In  that  way, 
words  sharing  a  similar 
initial  sequence  of  phonemes 
will  have  this  path  in 
common  in  the  tree.  This 
leads  to  memory  space 
savings  and  faster 
recognition. 

By  an  efficient  use  of 
subsets  of  the  vocabulary 
the  perplexity  of  the  system 
has  been  minimized. 

3.1.3.  The  best  results  of 
the  R&D  have  been  in  the 
area  of  wordspotting.  Single 
word  recognition  amounts  to 
spotting  the  word  out  of 
background  noise  represented 
by  garbage  word  models. 
Techniques  derived  from 
continuous  speech 
recognition  thus  proven  to 
be  useful  for  very  robust 
isolated  word  recognition 
(e.g.  the  algorithm  will 
discard  the  non-words  that 
are  typical  in  stress 
conditions) . 


3.2  Hardware. 

All  the  above  described 
functional  characteristics 
have  to  operate  in  real 
time.  To  this  end,  the 
algorithms  developed  in  C- 
language  on  SparcStations 
have  been  implemented  from 
the  signal  acquisition  upto 
the  decision  process  on  a 
development  DSP  board  (LSI 
TMS320  C30 ) .  The  host,  a  PC- 
AT  suffers  no  overload  due 
to  the  recognition  process 
since  all  computations  are 
performed  on  the  board. 
Floating  point  operations 
are  far  to  be  compulsory  for 
recognition,  except  for  the 
frequency  analysis  which 
becomes  time  consuming  when 
rescaling  steps  are 
required.  However,  the 


availability  of  an  efficient 
debugging  tool  influenced 
our  choice  of  the  board. 
Indeed,  real  time  processing 
requires  careful 
optimization  of  the 
assembler  code. 

Other  policies  can  be 
considered  where  the 
different  tasks  can  be 
shared  between  the  board(s) 
and  the  host.  For  instance, 
integration  of  voice  command 
in  an  electronic  copilot 
system  could  take  place 
amongst  many  other  pilot 
aids  and  be  implemented  on  a 
dedicated  host  while  some 
functions  only  (such  as 
anti-aliasing  filtering, 
sampling,  distance 
computations,  ...)  will  be 
executed  on  the  DSP  board. 


4.  FUTURE  R&D. 

Two  main  problems  are  still 
very  challenging  for 

embarked  voice  command 
systems . 

First,  the  recognition  score 
should  be  extremely  high  for 
meeting  pilot  requirements. 
This  implies  a  deep  study  of 
all  factors  able  to  improve 
robustness  :  more  sophis¬ 
ticated  models  without 

impairing  real  time 

processing,  lexicon  pre¬ 
processing,  (choice  of  low 
confusable  vocabulary  by 
using  a  confusion  table 
built  by  the  recognizer 

itself,  creation  of  multi¬ 
ple  phonetic  transcriptions 
without  using  expert 

knowledge,  . ..),  use  of 

HMM/neural  nets  hybrid 

approach,  determination  of 
robust  speech  features, 

word-spotting  techniques, 
introduction  of  spoken 

language  based  syntatico- 

semantic  constraints,  real 
time  coupling  with  existing 
signal  enhancement 
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techniques  (e.g.  microphone 
arrays,  . . . ) . 

Secondly,  the  system  should 
perfectly  match  the 
application  requirements. 
This  can  be  achieved  by 
carefully  tailoring  the 
dialogue  issues  and  using 
this  information  for 
controling  recognition. 
Moreover,  fast  training, 
description  of  the  lexicon 
in  subsets,  introduction  of 
new  words ,  speaker 
adaptation,  background  noise 
adaptation  and  acquisition 
channel  adaptation  should  be 
implemented  as  a  toolkit. 


5.  CONCLUSION. 

The  use  of  voice  technology 
as  an  alternative  MMI  opens 
extremely  interesting 
perspectives  and  will  enable 
the  pilot  to  interact  with 
his  aircraft  in  a  more 
efficient  way. 

However,  it  is  clear  that  we 
have  still  a  long  way  to  go, 
and  that  substantial 
research  remains  to  be 
undertaken.  Promising  is 
that  the  basic  technologies 
are  becoming  available  and 
that  the  researchers  have  a 
clear  view  of  the  direction 
future  research  should  take. 
And  who  knows,  we  may  fly 
our  helicopter  with  our 
voice  sooner  than  we 
expect  ! 
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Discussion 


QUESTION  C.  GULLI 

For  the  same  application,  what  are  the  possibilities  for  dependent  and 
independent  systems  working  at  the  same  time?  What  are  the  specific  fields 
of  each? 

REPLY 

If  the  question  refers  to  speaker  independent  systems,  the  answer  is 
that  the  ultimate  requirement  will  be  that  speaker  independent  recognisers 
should  be  used  although  acceptable  results  can  currently  only  be  obtained 
with  speaker  dependent  ones.  We  have  developed  a  technique  called  SPEAKER 
ADAPTATION  which,  within  a  few  minutes,  adapts  the  recognizer  to  the 
speaker's  characteristics,  i.e.  makes  it  speaker  dependent. 

If  the  question  refers  to  multiple  speakers,  speaking  a,,  the  same  time, 
and  injecting  this  mixed  signal  into  the  recognizer,  then  the  answer  is 
that  currert  algorithms  cannot  cope  with  this  scenario! 
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SOMMAIRE 

Lea  dispositifs  de  visualisation  du  futur  feront  appel  a  des 
systemes  muldmddia  permettant  un  dialogue  multimodal  entre 
l’homme  et  le  systeme. 


ARCHITECTURE  MATERIELLE 

Donnee  en  figure  1,  elle  est  constituee  des  trois  sous- 
ensembles  suivants:  les  medias  d’entree  ou  de  sortie,  le 
processeur  de  gestion,  la  source  d'images. 


Nous  dfecrivons  ici  le  concept  de  grand  6cran  interactif,  bad 
autour  d’un  mur  d' image  en  planche  de  bord  et  d'un 
ensemble  de  moyens  de  dialogues  utilises  simultanement.  II 
permettra  de  rfeduire  la  charge  de  travail  du  pilote  par  une 
optimisation  du  dialogue  avec  le  systeme  en  utilisant,  entre 
auties,  un  dispositif  intelligent  d’aide  au  dialogue. 

De  nombreuses  etudes  ont  fete  effectuees  sur  l'usage  isole  de 
dispositifs  d 'entree-sortie  (clavier,  souris,  manche, 
reconnaissance  vocale...)  (1),  mais  aucune  ne  fait  une 
approche  globale  du  dialogue  multimodal  dans  le  cockpit. 
Nous  avons  done  developpe  et  mis  en  oeuvre  un  dispositif 
experimental  pour  etudier  les  interactions  homme-systfeme 
utilisant.  l'oeil,  la  main  et  la  voix. 

INTRODUCTION 

Les  performances  du  couple  avion-pilote  dennent  pour  une 
bonne  part  a  la  qualite  du  controle  que  le  pilote  a  sur  le 
systeme:  le  pilote,  tout  en  recevant  1’etat  courant  du  systeme 
elabore  une  commande  a  laquelle  le  systeme  reagiL 
La  multiplicity  des  canaux  moteurs  et  des  canaux  sensoriels 
humains  autorise  leur  utilisation  simultanee  selon  diverses 
combinaisons;  cet  avantage  est  actuellement  peu  exploite. 

Le  systeme  d'interface  que  nous  decrivons  ici  est  bati  autour 
de  plusieurs  dispositifs  d’entree-sortie  exploitant  ces  differents 
canaux:  il  permet  l'enchainement  d’un  dialogue  suivant 
diverses  modalitfa.  Ce  dialogue  est  supervise  par  un  systeme 
de  gestion  intelligent. 

EVOLUTION  DU  COCKPIT 

Jusqu’k  ces  demi feres  annees,  l’interacuon  homme-machine 
s’effectuait  a  1‘aide  de  dispositifs  simples  a  modalite  unique 
(affichage  sur  cadran,  manche,  bouton  de  commande).  Les 
limites  de  cette  structure  de  cockpit  ont  rapidement  ete 
atteintes  au  fur  et  a  mesure  que  la  complexity  et  la  diversity 
des  taches  demandees  a  l’avion  de  combat  se  sont  accrues. 
Desormais,  le  pilote  dialogue  avec  un  systfeme  en  echangeant 
des  doimees,  e’est-a-dire,  des  entites  abstraites  II  sera  aide 
dans  sa  mission  par  un  copilote  electronique  (2). 

La  communication,  quant  a  elle,  a  besoin  d’une  interface 
adaptee,  telle  qu'un  grand  ecran  interactif;  celui-ci  permet 
rytablissement  d'un  dialogue  multimod;'  qui  se  rapproche, 
par  lk-meme,  de  la  communication  natureiio. 

Les  experimentations  preliminaires  (3,4)  que  nous  avoiis  deja 
menfees  sur  cette  interface  ont  montre  qu'une  analyse  des 
interactions  homme-systeme  est  indispensable  pour  crcer  une 
interface  optimum  de  dialogue  multimodal. 


Mfedlas  de  sortie 

-  Un  retroprojecteur  LCD  foumit  des  images  couleur  de  440 
x  480  pixels  sur  un  grand  ecran  de  520  x  400  mm2 
occupant  toute  la  planche  de  bord. 

-  Un  synthetiseur  vocal  (Datavox). 

Mfedtas  d’entrfee 

-  Un  oculomfetre  (NAC  EMR-V)  mesure  la  direction  du 
regard  par  rapport  a  la  tete  a  l’aide  d’une  micro-camera 
analysant  le  reflet  comeen  d’une  diode  infra-rouge  eclairant 
l’oeil  droiL 

-  Un  systfeme  de  reconnaissance  de  la  parole  en  continu 
(Datavox),  declenche  par  detection  d’activite,  effectue  une 
analyse  phonetique  et  syntaxique  du  signal  aprfes  l’avoir 
separe  en  messages  et  en  mots. 

-  Un  dispositif  a  fibres  optiques,  equipant  un  gant  porte  par 
la  main,  permet  de  mesurer  Tangle  de  flexion  des  deux 
premieres  articulations  de  chaque  doigt  (Data  Glove  de  VPL 
Research). 

-  Des  capteurs  electromagnetiques  (Polhemus  3  Space 
Isotrak)  couples  a  des  emetleurs  fixes  donnent  position  et 
orientation  de  la  main  et  de  la  tete. 


Figure  1;  Interface  pour  dialogue  multimodal. 
Processeurs  de  gestion 

-  La  gestion  des  medias  est  assuree  par  un  PC  386/20  MH/.; 
il  canalise  les  donnees  brutes  foumics  separement  par 
chaque  media,  el  dclivre  un  message  mubimcdia  au 
processeur  d;  gestion  du  dialogue. 

-  i^a  gestion  intelligente  du  dialogue  est  cffccluec  par  un 
autre  PC  386/25  MHz;  celui-ci  commande  la  source  d'images 
cl  le  synthetiseur  vocal. 
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Source  d’images 

La  station  de  travail  IRIS  4D25G  foumit,  au  retroprojecteur, 
les  images  de  type  TV  (figure  2).  Renouvelees  a  un  rythme 
dependant  de  leur  complexity  (environ  8  Hz),  elles  sont 
constitutes  pour  l’essendel: 

-  en  zone  centrale,  de  fenetres  variables  en  taille  et  en 
position,  chacune  contiert  une  figuration  avionique  de  type 
determint,  elles  peuvent  se  superposer  partiellement  ou 
total ement  suivant  plusieurs  plans. 

-  en  partie  inferieure,  d’etiquettes  representant  l’etat  des 
fenetres  (presence  a  I’ecran)  et  celui  des  medias  (marche, 
arret,  panne),  ainsi  qu’une  zone  de  securite  presentant  les 
fenetres  prioritaires,  en  medaillon. 


Figure  2:  Image  interactive  pr6sent6e  par  le  grand 
6cran  constituant  la  planche  de  bord. 

LES  ELEMENTS  DU  DIALOGUE 
Regard 

Sa  direction,  calculee  en  permanence  a  partir  de  l’orientation 
de  l’oeil  et  de  la  tele,  est  indiquee  par  un  symbole 
specifique,  mobile  sur  I’ecran;  i’activite  de  la  main  a  priorite 
sur  ce  canal. 

Vobt 

Le  vocabulaire  est  volontaircment  restreint  a  36  mots, 
regroupes  en  messages  de  1  a  3  mots. 

Main 

Les  4  postures  signifiantes  sont: 

-  "ddsigne":  index  tendu,  autres  doigts  replies. 

-  "o.k":  pouce  levt. 

-  "pris":  main  fermee,  doigts  tendus. 

-  "stop”:  main  ouverte,  pouce  replie. 

Le  suivi  de  designation  de  l’index  est  assure  par  un  symbole 
specifique  mobile  sur  1'ecr an 

Les  5  gestes  signifiants  sont: 

-  "prendre"-,  main  proche  de  I’ecran,  puis  se  refermant  en 

posture  "pris". 

-  "lacker":  quitter  la  posture  "pris". 

-  “ Jeter lac  her  apres  s'etre  eloigne  de  I’ecran. 

-  "rotation  droile “:  toumcr  la  main  de  30°  sur  elie-mcmc,  cn 
posture  "pris". 

-  "rotation  gauche":  toumer  la  main  dans  le  sens  inverse. 
Des  que  la  main  est  suffisamment  proche  de  I’ccTan,  un 
symbole  specifique,  donne  sa  position  courante. 


FINALITES  DU  DIALOGUE 

Les  actions  commandees  par  l’operateur  sont  actuellcment 
simples  et  en  nombre  reduit;  elles  portent, 

-  soil  sur  l'espace  graphique  lui-meme,  par  modifications  des 
fenetres:  presentation  a  l’ecran,  disparition,  deplacement, 
changement  de  taille,  mise  au  premier  plan, 

-  soit  sur  les  medias:  mise  en  marche,  arret. 

EXEMPLE  DE  DIALOGUE  MULTIMODAL 
Pour  commander  la  disparition  d’une  fenetre  deja  presente  a 
I’ecran,  l’operateur  dispose  de  5  modalites  differ entes  de 
dialogue  avec  le  systeme  (A:  action  operateur;  R:  reaction 
systeme);  la  fenetre  contient,  par  exemple,  une  figuration 
radar. 

1)  Verbalisation  totale: 

A=  voix:  "fermer  -  radar". 

R=  disparition  de  la  fenetre  radar  de  I’ecran. 

2)  Verbalisation  et  designation  manuelle  : 

A=  posture  de  main:  "designe". 

R=  apparition  du  symbole  de  designation. 

A=  mouvement  de  la  main. 

R=  suivi  du  mouvement  par  le  symbole. 

A=  index  designe  la  fenetre  radar  +  voix:  "fermer". 

R=  disparition  de  la  fenetre  radar. 

3)  Verbalisation  et  designation  au  regard: 

A=  mouvement  de  la  direction  du  regard. 

R  =  suivi  du  mouvement  par  le  symbole  du  regard. 

A=  regard  sur  la  fenetre  radar  +  voix:  "fermer". 

R=  disparition  de  la  fenetre  radar. 

4)  Manipulation  de  l’objet  graphique: 

A=  main  proche  de  i'ecran. 

R=  apparition  du  symbole  de  main. 

A=  mouvement  de  la  main  devant  I’ecran. 

R=  suivi  du  mouvement  par  le  symbole. 

t 

A=  main  face  a  la  fenetre  radar  +  geste  "prendre". 

R=  apparition  d’une  fenetre-squelette  sur  la  fenetre  radar. 

A=  mouvement  de  la  mail:  en  prehension  devant  l’ecran. 

R=  suivi  du  mouvement  par  le  squelette  et  par  le  symbole 
de  main. 

A=  eloignement  de  la  main  en  prehension. 

R=  disparition  du  squelette  et  du  symbole  de  main. 

A=  geste  "lacher”. 

R=  disparition  de  la  fenetre  radar. 

5)  Manipulation  d’une  metaphore  de  I'objct  graphique: 
A=  main  proche  de  I’ecran. 

R=  apparition  du  symbole  dc  main. 

A=  mouvement  dc  la  main  devant  I'ecran. 

R=  suivi  du  mouvement  par  Ic  symbole. 

A=  geste  "prendre"  sur  (’etiquette  representant  la  fenetre 
radar. 
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R=  apparition  d’un  symbole  circulaire  en  superposition  de 

l’etiquette. 

A=  geste  "rotation  gauche”. 

R=  changement  de  la  couleur  de  l’etiquette  et  disparition  de 
la  fenetre  radar. 

A=  geste  "lache”. 

R=  disparition  du  symbole  circulaire  sur  l'etiquette. 

LOGICIEL  DE  GESTION  DU  DIALOGUE 
Le  systkme  expert,  icrit  pour  l'essentiel  en  langage  Prolog  de 
fafon  k  faciliter  son  Evolution  en  phase  de  mise  au  point, 
constitue  le  coeur  du  traitement;  il  contient; 

-  un  module  de  comprehension  par  analyse  syntaxico- 
s  6m  antique  simple  du  message  multimedia  issu  du  serveur 
multimedia, 

-  les  rkgles  de  geslion  de  1’ application  qui  determinent  la 
validite  du  message  et  les  consequences  qu'il  induit  sur 
l’espace  graphique, 

-  les  regies  de  securite, 

-  la  base  de  fails  statiques  qui  comprend  les  lexiques  vocal 
et  gestuel,  la  position  par  defaut  des  fenetres,  les  fenetres 
priori  taires..., 

-  la  base  de  fails  dyn antiques  precisant,  par  exemple,  l’etat 
de  chaque  fenetre  (position,  taille,  presence  a  l'ecran),  la 
disponibilite  des  medias,  l'objet  designe,  la  position  initiate, 
la  taille  et  le  type  de  l'objet  manipuie,  la  proximite  de  la 
main  par  rapport  I  l’ecran  ... 

PERFORMANCE  ET  MULTIMODALITE 
Des  experimentations  preliminaires  ont  porte  sur  la  capacite 
d’utilisation  bimodale  de  ces  nouveaux  medias.  D'autres 
experimentations,  en  cours,  portent  sur  la  mesure  des 
performances  d’  un  dialogue  multimodal. 


CONCLUSION 

L'efficacile  du  controle  et  de  la  supervision  d’un  sysleme 
complexe,  tel  que  celui  des  avions  de  combat  modemcs  est 
fortement  dependant  de  la  structure  de  l'interface  homme- 
machine;  la  souplesse  que  lui  confere  le  caractcre 
multimodal  de  cclle  que  nous  avons  construite  permet  d’ 
adapter,  en  temps  reel,  le  mode  d'echange. 

Une  optimisation  de  l’efficacite  du  dialogue  est  ainsi  rendue 
possible. 
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SUMMARY 

The  integration  of  voice  into  a  complex  interface  like  that 
between  a  pilot  and  an  aircraft  is  not  trivial.  In  this  paper,  we 
try  to  address  some  of  the  factors  affecting  the  use  and 
integration  of  voice  in  human-machine  interfaces.  We  de¬ 
scribe  general  principles  for  merging  different  kinds  of 
human-machine  interaction,  and  apply  them  to  voice  interac¬ 
tion  in  the  cockpit.  We  do  this  despite  published  opinion  that 
psychological  principles  cannot  be  applied  in  the  design  of 
human-computer  interaction  (e.g.,  Landauer,  1991). 

The  theory  of  Layered  Protocols  (LP)  is  introduced  in 
context  of  the  more  general  Perceptual  Control  Theory  of 
behaviour  (PCT).  LP  theory  provides  a  model  for  describing 
interaction  between  complex  partners  based  on  a  layered 
structure  of  protocols  that  differ  in  levels  of  abstraction .  The 
proper  use  of  feedback  is  fundamental  to  both  LP  and  PCT. 

Voice  interaction  is  useful  mainly  for  the  control  of  tasks 
requiring  discrete  information.  Failure  of  voice  recognition 
systems  is  often  caused  by  inappropriate  feedback.  Provid¬ 
ing  feedback  and  forcing  correction  word  by  word  may 
increase  the  mental  load  on  a  user,  often  leading  to  instability 
in  the  interaction.  Such  inefficient,  and  often  frustrating,  use 
of  voice  interaction  can  often  be  overcome  throu  gh  the  use  of 
feedback  at  higher,  more  abstract,  layers  of  interaction. 
Successful  adoption  of  voice  interaction  depends  on  allocat¬ 
ing  the  appropriate  tasks  of  communication  to  the  voice 
protocol,  the  dynamic  modeling  of  the  partner,  and  the  use  of 
higher  level  protocols  to  help  control  potential  instability. 

INTRODUCTION 

Many  people  think  that  the  problem  of  getting  our  machines 
to  do  what  we  want  would  be  greatly  eased  if  we  could  talk 
to  them.  Experimental  aircraft  such  as  a  B  AC- 1 1 1  airliner  in 
the  UK  and  the  Mirage  IIIB  fighter  in  France  have  been 
provided  with  voice  I/O  for  this  reason.  But  not  all  attempts 
to  use  voice  have  been  as  effective  as  their  proponents  hoped. 

The  blame  for  failure  of  voice  I/O  is  often  placed  on  the  error 
rales  of  word  recognition,  but  other  factors  may  actually  be 
more  important.  For  example,  in  an  effort  to  allow  the  talker 
to  correct  recognition  errors,  the  designer  may  place  a  visual 
display  of  the  recognized  speech  somewhere  the  talker  can 
see  it.  But  if  there  are  only  a  few  errors  the  talker  may  well 
miss  them  on  the  display.  In  addition,  the  display  may  both 
distract  the  talker  from  other  visual  tasks  and  take  up  valu¬ 
able  space  in  the  cockpit.  Such  a  visual  read-back  of  the 
words  recognized  may  be  more  harmful  than  helpful  to  the 
use  of  voice  in  the  cockpit.  But  under  some  circumstances,  it 
could  be  exactly  what  is  needed. 


A  general  approach  to  communication 

The  theory  on  which  we  base  our  analysis  is  known  as  the 
Theory  of  Layered  Protocols  (LP;  Taylor,  1987,  1988a, 
1988b,  1989).  LP  is  a  general  theory  of  communication, 
compatible  with  an  approach  to  psychology  known  as  Per¬ 
ceptual  Control  Theory  (PCT;  Powers,  1973),  and  is  readily 
described  in  its  terms.  PCT  is  well  suited  to  the  description 
and  analysis  of  interactions  with  inanimate  objects,  whereas 
LP  emphasizes  the  mutuality  of  interaction  between  “intel¬ 
ligent”  partners,  and  can  be  seen  as  a  specialized  form  of 
PCT. 

The  theory  of  Layered  Protocols  (LP)  is  based  on  a  long 
tradition  in  psychology  that  people  perceive,  remember,  and 
act  at  a  number  of  levels  of  abstraction,  the  lower  levels 
supporting  the  higher.  LP  theory,  like  PCT,  is  based  on  the 
properties  of  a  hierarchy  of  control  systems,  but  focuses  on 
control  loops  that  incorporate  a  partner  with  some  independ¬ 
ence  of  perception  and  of  action. 

LP  theory  is  the  focus  of  this  paper,  but  before  discussing  it 
in  any  depth,  we  present  a  brief  introduction  to  PCT,  to  help 
lay  the  groundwork. 

Perceptual  control  theory 

According  to  PCT,  all  behavior  is  directed  to  the  control  of 
perceptions  at  a  variety  of  levels  of  abstraction  simultane¬ 
ously.  A  perception  might  be  the  tension  in  a  muscle  in¬ 
volved  in  the  turning  of  a  steering  wheel  that  allows  a  driver 
to  perceive  the  car  as  staying  on  the  road  during  a  trip  for 
money  for  the  purchase  of  food  that  allows  the  driver’s  body 
chemistry  to  stay  within  survivable  bounds. 

In  the  PCT  view,  all  living  things  consist  of  a  hierarchy  of 
control  systems  that  maintain  their  percepts  at  desired  levels 
by  means  of  actions  they  perform  on  the  outer  world.  In  very 
simple  organisms  such  as  bacteria,  there  may  be  only  one 
level  in  the  hierarchy,  and  only  one  possible  action  (e.g. 
“wiggle”),  but  this  suffices  to  move  the  bacterium  from 
nox  ious  en  vironments  into  suitable  ones  more  often  than  not. 
In  more  complex  organisms,  there  are  more  levels  of  control. 
Actions  are,  like  the  perceptions  they  control,  organized  at 
different  levels  of  abstraction.  The  driver  is  “doing”  many 
different  things  at  once:  following  a  life  plan,  performing  a 
job,  getting  wages,  driving,  following  a  bend  in  the  road, 
turning  the  steering  wheel,  and  tensing  certain  muscles, 
among  other  behaviors  (Vallacher  &  Wegner,  1987).  A 
feedback  loop  controls  each  perception  of  the  situation,  as 
shown  in  Figure  1. 
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Figure  1.  An  elementary  control  system  compares  its 
perceptual  input  with  a  desired  (reference)  percept,  and 
outputs  the  difference  as  an  error  signal  that  causes  effects 
to  happen  in  the  world.  These  effects  modify  the  perceptual 
signal  in  the  direction  of  the  desired  percept. 


An  elementary  control  system  (ECS),  such  as  shown  in 
Figure  1,  is  a  unit  of  a  control  hierarchy.  An  ECS  has  a 
percept  that  is  derived  by  some  function  from  the  sensory 
input  from  a  variety  of  sources,  including  lower-level  ECSs; 
it  has  a  reference  signal  derived  from  a  multiplicity  of  signals 
from  higher-level  ECSs,  and  it  has  a  error  signal  that  is  the 
difference  between  its  reference  and  its  percept.  Some  ampli¬ 
fying  operation  converts  the  error  signal  into  a  set  of  refer¬ 
ence  signals  for  lower  order  ECSs,  or,  at  the  lowest  level,  into 
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Figure  2.  An  Elementary  Control  System  is  normally  connected 
to  other  control  systems  at  both  higher  and  lower  levels, 
permitting  the  control  of  arbitrarily  complex  percepts. 


action  signals  for  effectors  (muscles),  as  shown  in  Figure  2. 
The  amplified  error  signal,  usually  caused  by  unpredicted 
events  in  the  environment,  leads  to  behavior  that  alters  the 
perceived  state  in,  the  protagonist  hopes,  the  desired  direc¬ 
tion. 

ECSs  are  connected  in  a  hierarchy  in  which  the  perceptual 
signals  of  low-level  control  systems  combine  to  provide  the 
sensory  inputs  to  higher  level  ones.  The  connections  among 
the  perceptual  elements  in  the  hierarchy  could  be  seen  as  a 
multi-layer  neural  network.  Such  a  hierarchy  could  develop 
perceptual  functions  of  arbitrary  complexity  (e.g.,  Lippmann, 
1987). 

In  the  same  way  that  the  perceptual  function  of  a  lower-level 
ECS  provides  one  of  the  inputs  to  the  perceptual  funtion  of 
a  higher  one,  so  do  the  action  outputs  of  high  level  ECSs 
combine  to  provide  the  reference  signals  for  low-level  ECSs. 
Figure  3  shows  a  sketch  of  such  a  hierarchy,  in  which  the 
lowest  ECSs  affect  and  sense  the  world  of  perceptible  things 
directly.  These  “perceptible  things”  include  the  aircraft  be¬ 
ing  flown,  as  well  as  communicative  partners. 

In  a  “classical”  PCT  hierarchy,  the  signals  (percept,  refer¬ 
ence,  and  error)  are  all  scalar.  If,  however,  one  looks  at  a  set 
of  ECSs  at  the  same  hierarchic  level  acting  in  parallel,  the 
effect  is  almost  the  same  as  if  there  were  one  ECS  with  a 
vector  percept,  reference,  and  error.  More  complex  struc¬ 
tures  for  the  signals  can  also  be  considered,  so  that  we  can 
talk  about  the  control  of  perceptions  of  arbitrary  complexity. 


In  the  hierarchic  control  system  depicted  in  Figure  3,  each  of 
the  control  systems  at  any  level  of  the  hierarchy  attempts  to 


Figure  3.  A  generic  view  of  hierarchic  perceptual  control 
structures  by  which  the  human  uses  behavior  to  control 
perceptions.  Higher  level  control  systems  receive  perceptual 
inputs  based  on  lower-level  perceptual  patterns,  and  provide 
outputs  corresponding  to  the  differences  between  what  they 
perceive  and  their  reference  percepts.  These  outputs  represent 
references,  or  goals,  for  lower  level  control  systems,  the 
lowest  of  which  control  the  effectors  that  act  on  the  world, 
changing  the  percepts  experienced  by  all  the  control  systems. 
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act  so  that  its  percept  agrees  with  its  reference  percept.  The 
higher  level  control  systems  do  this  by  adjusting  the  refer¬ 
ence  signals  sent  to  lower  level  systems.  The  lower  level 
control  systems  have  very  concrete  percepts  (and  references, 
therefore)  that  depend  directly  on  die  signals  impinging  on 
the  sensor  organs.  At  higher  levels,  the  percepts  represent 
more  abstract  structures,  not  directly  observable  by  the  sense 
organs. 

One  can  see  a  structured  control  system  as  a  goal-seeking 
device.  It  works  toward  ever  changing  goals  defined  by  its 
structured  reference  signals.  For  example,  the  reference 
percept  for  a  pilot  may  be  the  perception  of  a  safe  landing  at 
the  destination  airport.  The  achievement  of  that  goal  in¬ 
volves  setting  several  other  reference  percepts  for  lower- 
level  ECSs,  such  as  the  perception  of  passing  a  given  se¬ 
quence  of  waypoints,  the  perception  of  Hying  at  a  particular 
altitude,  and  so  forth. 

Communication 

In  the  framework  of  either  Perceptual  Control  Theory  or 
Layered  Protocol  theory  there  is  no  structural  difference 
between  controlling  an  inanimate  tool  and  communicating 
with  an  intelligent  partner.  A  communicative  partner  is  part 
of  the  perceptible  outer  world  that  is  affected  by  behavior. 
B  ut  unlike  a  tool,  an  “intelligent”  communicative  partner  has 
its,  his,  or  her  own  control  systems  that  perceive  and  act  We 
give  the  name  “virtual  messages”  to  communications  be¬ 
tween  the  high-level  structures  of  the  protagonist  and  corre¬ 
sponding  ones  of  the  partner. 

We  take  communication  to  cover  a  wide  range  of  interaction, 
from  simple  use  of  a  tool  at  one  pole  of  a  continuum,  to 
complex  discussions  such  as  philosophical  argument  at  the 
other.  We  describe  both  sorts  of  interaction  in  the  same  way, 
using  the  same  principles,  but  with  great  differences  in  the 
quantitative  aspects  of  the  interaction  structure. 

Whether  the  partner  is  a  machine  or  a  person,  the  protagonist 
performs  some  action  that  affects  the  partner’s  behavior  in 
some  detectable  manner,  as  shown  in  Figure  4.  A  pilot  may 
pull  back  on  the  stick,  and  the  partner  (the  plane)  begins  to 
climb;  or  a  philosopher  may  present  an  argument  to  which 
the  partner  responds  with  a  counter-argument.  In  either  case, 
the  fundamental  construct  is  a  feedback  loop.  If  the  climb  is 
too  steep  or  not  steep  enough,  the  pilot  changes  the  angle  of 
the  stick;  the  philosopher  changes  some  part  of  the  argument, 
augmenting,  explaining,  correcting  in  a  way  calculated  to 
bring  the  partner  to  agreement.  In  either  case,  the  protagonist 
behaves  in  such  a  way  as  to  perceive  a  desired  situation. 

Using  a  different  terminology,  the  protagonist  has  a  percep¬ 
tual  goal  that  may  be  achieved  through  behavior.  The  success 
of  the  behavior  in  bringing  the  goal  closer  is  monitored,  and 
the  behavior  altered  accordingly.  If  the  achievement  of  the 
goal  involves  communicating  with  a  partner,  there  is  a 
communicative  goal,  which  may  be  achieved  through  either 
dialogue  or  non-dialogue  methods,  depending  on  the  pro¬ 
tagonist’s  model  of  the  partner  and  on  other  circumstances, 
such  as  whether  the  protagonist  wants  the  partner  to  realize 
that  the  communication  is  happening,  or  whether  the  partner 
is  even  capable  of  engaging  in  dialogue. 


There  is  a  critical  difference  between  using  a  tool  and 
communicating  with  an  intelligent  partner.  The  tool  has  no 
goals  of  its  own — no  reference  states  that  it  uses  to  set  desired 
percepts  that  might  conflict  with  those  desired  by  the  pro¬ 
tagonist.  The  tool  is  neither  cooperative  nor  antagonistic, 
whereas  a  human  communicative  partner  may  be  either,  but 
is  unlikely  to  be  passively  neutral  in  respect  of  the  commu¬ 
nication.  The  tool  is  reactive,  whereas  the  person  may  be 
proactive.  An  “intelligent”  machine  falls  somewhere  be¬ 
tween,  in  that  it  can  be  controlling  sensory  inputs  to  accord 
with  references  that  are  set  independently  of,  and  may  be 
unknown  to,  the  human  user.  A  computer-controlled  aircraft 
that  has  flight  envelope  limits  built  into  its  program  may 
“deliberately”  thwart  the  intentions  of  a  pilot  to  perform 
some  manoeuvre.  Such  a  machine  is  the  kind  of  partner  that 
we  consider  in  this  paper. 

Background  to  Layered  Protocol  theory 

The  principles  we  will  be  describing  are  part  of  the  Layered 
Protocols  (LP)  theory  of  communication  between  “intelli¬ 
gent”  entities.  In  this  context,  “intelligent”  implies  not  clev¬ 
erness  so  much  as  a  degree  of  independence  between  the 
entities  in  three  respects: 

•  independence  of  design, 

•  independence  of  sensing  mechanism,  and 

•  independence  of  action. 

Independence  of  design  means  that  neither  partner  can  be 
sure  precisely  how  the  other  will  interpret  any  specific 
communication.  Independence  of  sensing  mechanism  means 
that  neither  partner  can  know  exactly  what  information  the 
other  has  available.  Independence  of  action  means  that 
neither  partner  can  know  at  any  moment  all  of  what  the  other 
is  doing.  Independence  of  action  lies  at  the  heart  of  the  link 
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Figure  4.  The  formal  similarity  between  controlling  the  attitude 
of  an  aircraft  and  communicating  a  persuasive  argument.  In 
either  case,  the  important  issue  is  whether  the  protagonist  (on 
the  left,  in  each  part  of  the  figure)  can  come  to  perceive  that 
the  effect  on  the  partner  is  as  desired. 
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between  “intelligence”  and  independence,  because  it  is  the 
ability  of  an  entity  to  perform  actions  that  hear  a  useful 
relation  to  its  circumstances  that  normally  leads  one  to 
classify  it  as  “intelligent.” 

Communications  between  a  pilot  and  an  aircraft  largely  fulfil 
the  three  independence  criteria,  and  will  increasingly  do  so 
as  computerization  of  aircraft  proceeds.  The  aircraft  and  the 
pilot  clearly  were  not  designed  together,  the  aircraft  has 
many  sensing  systems  not  directly  accessible  to  the  pilot,  and 
vice-versa;  and  the  aircraft  can  perform  actions,  such  as 
manipulating  information,  that  are  not  known  in  detail  to  the 
pilot.  As  a  trivial  example,  when  the  aircraft  is  on  autopilot, 
the  pilot  may  not  even  be  aware  of  some  of  the  flight  events 
it  performs. 

The  independence  criteria  mean  that  neither  communicating 
partner  can  be  aware  of  the  exact  effect  of  any  particular 
communication.  Under  these  circumstances  no  message  can 
be  encoded  with  sufficient  redundancy  to  guarantee  error- 
free  reception,  in  contrast  to  the  classical  situation  in  which 
the  only  communicative  difficulty  is  noise  in  the  communi¬ 
cation  channel  and  reception  delay  associated  with  error- 
correction  coding.  With  independence,  the  receiving  partner 
must  provide  the  transmitter  with  feedback  that  helps  each 
partner  to  believe  that  the  intended  message  was  the  one 
received,  and  to  correct  the  situation  if  it  was  not 

When  feedback  enters  the  situation,  questions  of  stability 
arise.  If  the  feedback  is  delayed,  has  too  strong  an  influence 
on  the  forward  channel,  or  changes  too  rapidly,  the  whole 
connection  may  be  dominated  by  instability  in  the  feedback 
loop,  to  the  exclusion  of  the  information  that  is  supposed  to 
be  transmitted.  The  stability  criteria  for  feedback  systems 
would  oppose  those  for  rapid  and  effective  information 
transmission,  if  a  single  error-correcting  feedback  loop  were 
used  (Taylor,  1989).  But  the  effects  of  the  opposition  can  be 
reduced  by  dividing  the  work  among  a  hierarchy  of  feedback 
systems,  each  of  which  occur  over  successively  longer  time 
scales.  In  voice,  for  example,  corrections  can  be  made  at  the 
level  of  word  (“what  word  was  that?’),  proposition  (“You 
mean  alter  the  map  display?”),  or  higher-level  constructs 
(“You  mean  to  suggest  that  we  will  have  very  little  fuel 
reserve?”). 

Before  considering  the  Layered  Protocol  theory  in  more 
detail,  let  us  discuss  some  considerations  relating  to  the  voice 
channel  through  which  much  human-human  (and  very  little 
human-machine)  communication  is  passed. 

Voice  in  aircraft 

What  kind  of  material  is  suited  for  voice  input?  In  human 
communication,  voice  is  used  to  communicate  subtle  rela¬ 
tionships  among  a  wide  range  of  concepts,  a  range  much 
wider  than  can  be  accommodated  by  gesture  or  by  any  other 
means  of  communication,  except  possible  gestural  language 
such  as  American  Sign  Language.  Only  writing  offers  a 
comparable  range  of  possibilities  for  communication,  and  it 
is  restricted  by  its  inability  to  convey  nuances  of  affect. 

In  communicating  with  a  machine,  voice  cannot  be  used  in 
its  “natural”  function,  for  two  reasons.  Firstly,  current  and 


projected  recognizers  have  neither  the  range  of  vocabulary 
nor  the  ability  to  deal  with  intonational  modulation  de¬ 
manded  by  normal  conversation;  secondly,  the  machines 
with  which  we  might  wish  to  communicate  do  not  have  the 
intelligence  to  interpret  the  kind  of  subtleties  that  humans  are 
accustomed  to  conveying  by  voice.  In  dealing  with  ma¬ 
chines,  humans  must  use  voice  in  an  unnatural  way,  restrict¬ 
ing  the  vocabulary  and  relying  on  words  by  themselves  to 
convey  the  intent  of  the  communication.  Indeed,  voice 
recognition  by  machine  has  ordinarily  been  taken  to  be  a 
problem  in  word  recognition,  and  only  occasionally  until 
recently  has  there  been  any  significant  interest  in  speech 
understanding  (e.g.  Klatt,  1977).  Why,  then,  should  anyone 
want  to  use  voice  in  interacting  with  machines? 

Fly  ing  an  aircraft  is  fundamentally  a  question  of  making  it  go 
where  the  pilot  wants  it  to  go,  and  perhaps  to  perform  other 
manoeuvres  while  it  is  doing  so.  From  moment  to  moment, 
going  where  the  pilot  wants  is  a  problem  in  continuous 
control,  but  on  a  longer  time  scale  the  problem  is  discrete — 
the  plane  should  arrive  at  this  airport  or  that,  maintain  this 
altitude  or  that,  pass  through  this  waypoint  or  that.  Voice  is 
adequate  for  communicating  discrete  information,  but  not 
for  continuous  control.  It  follows,  then,  that  the  tasks  for 
which  voice  should  be  used  are  not  those  in  which  the  pilot 
needs  continuous  control.  It  also  follows  that  the  tasks  that 
use  voice  must  be  integrated  with  those  under  continuous 
pilot  control,  and  in  an  “intelligent”  future  aircraft  with  tasks 
controlled  by  largely  autonomous  subsystems. 

Voice  is  not  the  only  means  by  which  the  pilot  can  commu¬ 
nicate  discrete  symbolic  information  to  the  aircraft.  Voice 
must  compete  with  pushbuttons,  keyboards  both  soft  and 
hard,  and  even  possibly  with  voice  communication  to  other 
humans.  If  there  are  few  choices  and  very  clean  error-free 
communication  is  required,  a  pushbutton  may  be  more 
appropriate  than  voice,  provided  that  the  pilot  has  a  hand  free 
to  push  it,  and  perhaps  eyes  free  to  see  it  pushed  correctly. 

One  may  not  want  to  use  voice  when  the  timing  of  an  event 
is  crucial.  Fingers  are  much  better  than  voice  for  exactly 
timed  events.  It  may  be  appropriate  to  arm  a  weapon  by 
voice,  but  not  to  activate  it.  An  error  in  arming  can  be 
retracted,  but  a  gun  cannot  be  unshot,  or  a  bomb  undropped. 
A  weapon  used  at  a  time  200  msec  from  optimum  might  as 
well  not  have  been  used.  Even  in  far  future  aircraft  with 
excellent  voice  recognition  capacities,  it  is  unlikely  that 
spatially  discrete  buttons  and  switches  will  be  totally  super¬ 
seded  by  voice. 

What,  then,  is  voice  good  for?  Primarily  for  strategic  infor¬ 
mation,  communication  that  is  not  time-critical,  the  control 
of  other  interactions  and  displays,  and  in  general  those  things 
that  involve  selection  among  many  discrete  possibilities  that 
can  combine  in  different  ways  and  in  which  errors  are 
recoverable.  Voice  is  particularly  useful  when  the  hands  and 
eyes  are  otherwise  occupied,  and  this  is  the  main  reason  for 
wanting  voice  in  the  cockpit  at  all. 

Voice  recognition  equipment  has  its  limitations,  especially 
in  complex,  rapidly  changing  situations  such  as  are  encoun¬ 
tered  in  an  aircraft  cockpit.  Natural  human  voice  communi¬ 
cation  is  less  syntactically  constrained  than  is  written  text, 
but  recognizers  for  continuous  speech  usually  demand  ad- 
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herence  to  a  fairly  rigid  syntax  as  the  price  of  recognizing  an 
adequate  vocabulary.  In  stressful  situations,  people  may 
forget  to  use  the  correct  highly  constrained  syntax.  On  the 
other  hand,  it  has  been  reported  that  in  emergency  conditions 
when  the  pilot  has  a  chance  of  landing  safely,  the  language 
may  revert  to  the  correct  form,  perhaps  because  it  takes  less 
cognitive  resources  to  use  the  highly  trained  forms  than  to 
invent  new  dialogue.  Even  in  conditions  of  relatively  low 
stress,  such  as  normal  air  traffic  control  (ATC)  interactions, 
pilots  and  controllers  often  go  well  beyond  the  bounds  of  the 
prescribed  syntax,  and  this  has  presented  significant  prob¬ 
lems  to  researchers  attempting  to  use  voice  recognition  as  a 
component  in  training  systems  for  ATC  (F.  N6el,  Personal 
Communication,  April  1990). 

Speech  recognizers  tend  to  have  trouble  during  conditions  of 
stress,  although  the  few  existing  studies  of  stressed  voice 
show  no  consistent  trends  in  the  parameters  of  speech  (C. 
Weintstein,  Personal  Communication,  April  1992).  It  is 
probable  that  the  stressed  voice  is  more  variable  than  the 
unstressed  voice,  and  this  causes  recognition  errors .  B  ut  it  is 
particularly  in  stressful  conditions  that  the  accuracy  of  the 
recognizer  is  most  important. 

People  tend  to  stop  talking  while  they  deal  with  immediate 
control  problems,  so  it  is  likely  that  voice  is  best  used  for 
tasks  that  can  be  postponed  until  immediate  stresses  have 
passed,  or  that  involve  preparation  for  later  periods  of  stress. 
This  will  not  necessarily  be  true  if  the  use  of  voice  is  part  of 
the  trained  behaviour  to  deal  with  the  stress  condition,  but  it 
otherwise  imposes  some  restrictions  on  the  machine  side  of 
the  interface,  because  synchronization  between  voice  and 
other  controls  may  not  be  reliable. 

Voice  in  the  cockpit  faces  a  hurdle  not  placed  in  the  path  of 
traditional  means  of  communication  between  the  pilot  and 
his  aircraft.  Voice  is  psychologically  required  to  be  error- 
free,  whereas  keyboard  entry  for  the  same  task  is  not  Why? 
Could  it  be  that  error  detection  and  recovery  is  thought  to  be 
easier  and  quicker  for  keyboard  entry  than  for  voice?  In  fact 
voice  entry  for  numbers  may  well  be  more  accurate  than 
through  a  keyboard,  even  under  office  conditions  in  which 
keyboard  entry  is  easy.  Or  is  it  that  the  pilot  knows  that  the 
keyboard  will  accurately  report  the  character  corresponding 
to  the  pressed  key,  whether  the  key  be  right  or  wrong, 
whereas  the  voice  recognizer  will  sometimes  report  the 
wrong  word  even  when  the  pilot  speaks  the  correct  one?  In 
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Figure  5.  A  Protocol  converts  information  to  be  transmitted 
into  a  less  abstract  form  that  is  communicated  to  the  partner. 
The  protocol  may  require  feedback  to  ensure  correct 
transmission,  but  the  virtual  message  goes  only  one  way,  and 
is  interpreted  by  the  recipient  at  the  same  level  of  abstraction 
as  the  original  information  that  was  to  be  transmitted. 


military  action,  operators  (pilots)  must  have  confidence  in 
their  abilities  and  their  equipment,  whether  or  not  either  be 
warranted. 

STRUCTURE  OF  A  MULTIMODAL  INTERFACE 

To  see  where  voice  fits  into  a  Layered  Protocol  structure,  we 
must  first  deal  with  the  general  question  of  how  the  theory  of 
Layered  Protocols  can  be  used  to  describe  a  complex  inter¬ 
action  between  “intelligent”  partners.  The  principles  are 
quite  general .  To  apply  them  to  specific  situations,  the  details 
must  be  filled  in,  but  the  structure  remains  the  same.  The 
Layered  Protocol  structure  we  describe  here  is  somewhat 
evolved  from  that  presented  by  Taylor  (1988a,  b;  1989),  but 
most  of  what  was  described  there  is  still  valid. 


Support  structure 

A  Layered  Protocol  interface  has  two  important  aspects:  the 
protocols,  and  the  layered  support  structures  that  link  the 
protocols.  Let  us  first  briefly  consider  the  support  structure, 
before  we  deal  in  more  detail  with  the  nature  of  protocols.  For 
now,  assume  that  a  protocol  is  a  means  whereby  a  chunk  cf 
information  to  be  communicated  is  transformed  into  some 
less  abstract  form  by  the  transmitting  partner,  and  interpreted 
by  the  receiving  partner,  possibly  after  much  feedback  and 
correction,  into  a  similarly  abstract  (though  possibly  differ¬ 
ent)  form,  as  shown  in  Figure  5.  The  “chunk”  might  be  the 
information  whose  transmission  constitutes  the  original  com¬ 
municative  goal  of  the  transmitter,  or  it  might  be  the  result  of 
a  transformation  performed  by  a  higher-level  protocol  as 
shown  in  Figure  6.  In  either  case,  the  result  of  the  transforma¬ 
tion  performed  by  the  protocol  is  a  “virtual  message”  that 
passes  between  the  two  parties.  Only  when  the  successive 
transformations  have  arrived  at  physical  phenomena  such  as 
sound  waves  or  photon  streams  can  the  messages  be  termed 
“real.” 

Virtual  messages  are  realized  by  means  of  actions  that  may 
take  different  forms  under  different  circumstances;  the  same 
message  may  be  transmitted  by  voice  on  one  occasion,  by 
gesture  on  another,  and  on  a  third  by  a  combination  of  the 
two.  It  does  not  matter  which  mode  is  used,  provided  that  the 
result  satisfies  the  protagonist’ s  requirement  to  perceive  that 
the  virtual  message  was  received.  It  is  the  beliefs  or  percepts 
of  the  originator,  not  the  recipient,  that  determine  the  need 
for,  the  means  of,  and  the  success  of,  the  communication. 


*  / 

Figure  6.  The  information  that  is  transmitted  by  a  protocol  may 
be  the  virtual  message  of  a  higher  level  protocol.  The  lower- 
level,  less  abstract,  protocol  is  said  to  support  the  higher-level 
one.  Feedback  connections  are  important  at  both  levels,  but 
are  not  shown  in  this  figure. 
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Those  beliefs  are  most  strongly  affected  by  feedback  from 
the  recipient,  and  it  is  the  highest  level  of  feedback  that 
matters  most  If  high-level  control  systems  are  satisfied, 
lower-level  ones  need  not  do  anything.  But  higher-level 
systems  are  more  likely  to  be  satisfied  if  lower-level  ones 
perform  properly. 

Sometimes  information  is  more  easily  transmitted  in  one 
form,  say  verbal,  and  sometimes  in  another,  say  pictorial.  Or 
perhaps  part  of  a  higher-level  message  may  be  transformed 
into  words,  and  another  part  into  pictures,  as  happens,  for 
example,  with  labels  on  a  map.  In  this  situation,  two  protocols 
support  the  transmission  of  a  single  chunk  of  information,  as 
in  Figure  7.  Such  an  arrangement  is  known  as  “diviplexing.” 
The  converse,  multiplexing,  also  is  common;  one  lower 
protocol  supports  two  higher  ones  simultaneously.  Multiple 
windows  on  a  computer  screen  provide  a  well-known  exam¬ 
ple.  The  screen  is  a  single  visual  display  channel,  but  differ¬ 
ent  processes  communicate  with  the  user  through  it,  distin¬ 
guishing  their  outputs  by  locating  them  within  different 
windows  on  the  screen  surface. 

The  whole  structure  of  support  within  the  network  is  an 
acyclic  graph,  in  which  converging  and  diverging  threads  of 
support  allow  multiple  kinds  of  high-level  messages  to  be 
communicated  through  a  variety  of  physical  media.  Virtual 
messages  are  transformed  and  transformed  again,  into  ever 
less  abstract  forms,  before  being  physically  transmitted  to 
the  partner,  where  they  are  reassembled  and  interpreted. 

An  important  issue  is  how  the  partner  can  determine  the 
interrelations  of  the  partial  messages  being  transmitted  by 
the  various  differentmeans.  What  signals  how  twodiviplexed 
messages  ought  to  be  recombined,  or  in  a  multiplexed 
message  how  the  components  with  different  destinations 
should  be  identified?  The  necessary  information  must  be  in 
one  of  two  places:  in  the  message  forms,  or  in  specific  prior 
knowledge  of  the  receiving  partner.  If  it  is  in  the  message 
form,  then  it  is  part  of  the  protocol  for  that  type  of  message, 
as,  for  example,  in  the  protocol  that  whatever  appears  within 
the  frame  of  one  window  on  the  screen  must  belong  to  a 
single  task. 

Protocol  Structure 

A  protocol  has  a  simple  job:  to  take  as  a  goal  the  transfer  of 
some  chunk  of  information  from  one  partner  to  the  other  by 


Figure  7.  it  is  possible  for  a  higher-level  protocol  to  be 
supported  by  more  than  one  lower-level  protocol.  This 
arrangement  is  called  “duplexing.’  At  th9  higher  level, 
information  may  be  transmitted  in  the  form  of  a  map,  for 
example,  which  at  a  lower  level  is  transmitted  partly  in 
pictorial  and  partly  in  verbal  form.  Feedback  connections  are 
omitted  for  clarity  in  this  figure. 


transforming  it  into  some  less  abstract  form  that  the  protocol 
is  able  to  transmit.  The  protocol  is  executed  by  a  pair  of 
“protocol  nodes,”  a  transmitting  node  in  the  originating 
partner,  and  a  receiving  node  in  the  recipient  partner.  The 
protocol  may,  but  need  not,  take  advantage  of  situation- 
specific  information  such  as  information  about  the  partner, 
the  task,  the  local  state  of  the  world,  the  recent  and  planned 
dialogue,  and  so  forth.  In  its  simplest  form,  the  protocol  may 
simply  translate  the  originator’s  intent  to  pass  a  character, 
say,  into  the  depression  of  a  key  on  a  keyboard,  which  is 
directly  interpreted  by  the  recipient  (computer)  as  the  trans¬ 
mission  of  the  character.  Similar  simplicity  would  attend  the 
transmission  of  a  spoken  word  through  a  very  reliable 
recognizer  that  gave  no  feedback  (if  such  a  thing  were  to 
exist);  the  user  would  speak  the  word  (translating  it  into  a 
pattern  of  acoustic  waves),  and  the  recognizer  would  recon¬ 
vert  it  into  a  computer  representation  of  the  same  word  with 
no  further  ado. 

Such  error-free  transmission  is  rare,  and  perhaps  impossible 
even  between  partners  that  lack  the  three  independences 
(Elias,  1953).  People  do  mistype,  hitting  the  wrong  key  even 
though  they  know  which  one  they  intend  to  strike;  speech 
recognizers  do  make  errors,  even  though  the  user  speaks  the 
correct  word  clearly.  In  either  case,  if  the  errors  are  not 
detected  and  corrected,  higher-level  protocols  must  be  de¬ 
signed  to  accept  the  erroneous  keystroke  or  word,  and  to 
perform  acceptably  in  spite  of  the  error.  The  options,  then, 
are  to  correct  errors  before  they  are  taken  as  truth  by  higher- 
level  protocols,  or  to  accept  that  errors  will  be  made,  and  to 
ensure  that  the  higher-level  protocols  are  flexible  enough  to 
handle  them. 

People  do  not  normally  speak  with  the  expectation  that  then- 
partner  will  query  each  dubious  word.  They  expect  that  the 
partner  will  try  to  make  sense  of  what  they  are  saying,  despite 
the  fact  that  many  words  are  slurred,  shortened,  or  even 
omitted.  Think  of  the  typical  American  pronunciation  of  the 
word  “President,”  as  in  “Pres’n  Bush.”  The  word  “and” 
usually  is  pronounced  as  “n”  if  it  is  not  acoustically  omitted 
entirely.  Such  “flaws”  at  the  word  level  cause  nodifficulty  to 
a  fluent  English  speaker  listening  to  them,  though  they  may 
well  make  it  impossible  for  a  less  fluent  listener  to  under¬ 
stand  the  speech  at  all.  The  internal  redundancies  in  the 
structure  of  the  sentences  allow  the  missing  or  deficient 
words  to  be  reconstituted  with  almost  total  accuracy.  Listen¬ 
ers  rarely  are  even  aware  that  the  reconstruction  has  oc¬ 
curred,  unless  they  listen  very  critically.  Word-by-word 
feedback  of  what  the  recognizer  has  recognized  is  therefore 
not  likely  to  be  part  of  a  good  set  of  protocols  for  dealing  with 
voice  input,  unless  the  recognizer  performance  is  very  poor. 

A  second  conclusion  about  these  simple  protocols  without 
feedback,  imphcit  in  the  foregoing  discussion,  is  that  the 
results  are  almost  never  used  directly,  but  are  part  of  a  larger 
message.  It  is  true  that  in  a  simple  tool-using  protocol  the 
push  of  a  button  is  the  signal  for  something  to  happen,  often 
independently  of  any  prior  tool  manipulation,  but  even  there, 
the  effective  action  is  often  part  of  a  larger  sequence.  Feed¬ 
back  can  be  omitted  if  and  only  if  (a)  the  message  is  with  high 
probability  going  to  be  properly  interpreted,  and  (b)  the 
message  will  be  incorporated  into  a  higher-level  message 
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that  does  have  feedback,  or  will  be  used  in  an  observable  and  of  tbe  spoken  material,  but  is  more  likely  to  be  an  entirely 

correctable  way  to  affect  the  outer  world.  different  kind  of  presentation. 


What  is  the  point  of  feedback?  To  understand  this,  we  must 
go  further  into  tbe  whole  process  of  action  and  communica¬ 
tion. 

Feedback 

A  protocol  node  has  functions  quite  analogous  to  those  of  an 
ECS,  as  shown  in  Figure  8.  The  protocol  node  in  the  origina¬ 
tor  has  a  goal  that  the  recipient’s  receiving  node  should  have 
some  chunk  of  information,  and  a  reference  belief  about  what 
information  the  partner  has;  it  performs  an  action  in  the  form 
of  a  virtual  message  intended  to  affect  the  partner’ s  belief  in 
the  desired  direction,  and  acquires  feedback  that  allows  it  to 
modify  its  beliefs  and  perhaps  alter  the  way  it  presents  the 
virtual  message.  Parallel  relationbships  hold  for  the  receiv¬ 
ing  node. 

The  key  to  the  action  of  the  transmitting  protocol  node  is  in 
the  relation  between  the  Model  on  the  one  hand  and  the  Coder 
and  Decoder  on  the  other.  The  Model  is  the  analogue  to  the 
comparator  of  the  ECS.  It  compares  the  chunk  of  informa¬ 
tion  to  be  sent  with  that  believed  to  be  known  to  the  recipient. 
How  complex  is  the  comparison  that  leads  to  the  output 
message?  At  very  low  levels,  the  Model  is  vestigial,  and  the 
Coder  and  Decoder  do  all  the  work.  By  design,  a  low  level 
protocol  node  always  assumes  that  the  partner  does  not  have 
the  information,  so  the  virtual  message  carries  it  For  exam¬ 
ple,  if  a  keyboard  command  requires  the  letter  “p,”  the  user 
presses  the  “p”  key.  All  of  the  behavior  of  such  a  low-level 
protocol  node  is  determined  by  the  Coder  (or,  in  a  receiving 
node,  by  the  Decoder). 

In  contrast,  at  very  high  levels,  almost  all  the  work  is  done  in 
the  Model,  and  the  Coder  and  Decoder  are  relatively  simple. 
At  these  levels,  the  partner  probably  already  knows  much  of 
the  information  that  the  originator  wishes  to  get  across,  or  at 
least  has  analogous  information  that  can  be  used  to  assist  in 
the  encoding  process. 

The  question  of  what  the  partner  knows  or  ought  to  know  is 
crucial  in  determining  the  need  for  feedback.  From  the 
viewpoint  of  the  recipient,  if  the  originator  probably  believes 
that  the  message  has  been  successfully  received,  there  is  no 
need  to  provide  feedback.  On  the  other  hand,  if  the  recipient 
believes  that  originator  quite  probably  is  uncertain  what 
message  was  received,  then  informative  feedback  is  re¬ 
quired.  As  Pask  (e.g.  Pask,  1980)  points  out,  informative 
feedback  should  normally  not  be  a  mere  echo  of  the  overt 
content  of  the  message.  Rather,  it  should  be  some  reinterpre¬ 
tation  of  the  message  in  a  different  form  that  the  originator 
can  recognize  as  having  the  same  intention. 

For  voice,  the  worst  feedback  probably  is  a  vocal  echo,  which 
is  subject  to  a  set  of  error  probabilities  very  like  those  that 
applied  to  the  original  vocal  message,  so  that  the  speaker  may 
well  misinterpret  the  echo  of  a  wrong  recognition  as  being 
correct.  What  kind  of  feedback  is  best  for  voice  will  depend 
on  the  situation  and  the  probability  of  error  in  the  voice 
recognition  equipment.  It  might  even  be  a  vocal  paraphrase 


As  an  example  of  appropriate  feedback,  consider  the  use  of 
voice  to  control  a  map  display  of  variable  scale  that  shows  the 
region  centred  on  the  aircraft,  such  as  in  the  experimental 
B  AC-1 1 1  flown  by  RAE  Bedford.  In  this  aircraft,  the  pilot 
can  issue  a  command  something  like  “Map  range  fifty  miles” 
to  set  the  range  of  the  outer  ring  of  the  map.  The  experimental 
recognition  system  incorporates  a  visual  display  that  shows 
the  words  recognized,  but  this  display  is  ordinarily  used  only 
by  pilots  for  whom  recognition  performance  is  bad  (Modera¬ 
tor’ s  comment  in  Taylor,  1987).  A  pilot  for  whom  recogni¬ 
tion  is  normally  good  discovers  whether  a  particular  recog¬ 
nition  was  correct  through  the  action  of  the  map  display, 
which  may  change  to  the  wrong  range,  if,  for  example, 
“fifty”  is  mistaken  for  “fifteen.”  In  such  a  case,  the  pilot 
merely  corrects  the  effect  of  the  message  by  using  a  new 
command,  perhaps  repeating  the  original.  The  error  would 
not  be  repeated  if  the  system  were  designed  to  make  the 
pragmatic  assumption  that  a  “map  range”  command  is  in¬ 
tended  to  change  the  range. 


The  primary  reason  for  introducing  voice  into  the  cockpit  is 
supposed  to  be  to  reduce  the  load  on  the  pilot’ s  resources,  not 
to  give  him  more  capability.  After  all,  there  is  very  little  that 
voice  can  do  that  cannot  be  done  tactually.  However,  keying 
a  number  can  distract  the  pilot  from  flying  the  aircraft  more 
than  does  saying  the  number.  In  low-level  flight,  it  is  impor¬ 
tant  that  the  pilot  fly  accurately,  and  voice  is  therefore  a 
desirable  means  of  entering  numbers,  provided  the  numbers 
are  usually  recognized  correctly.  But  if  the  pilot  is  not 
assured  that  the  number  will  be  assigned  to  the  proper 
funcdon,  the  use  of  voice  to  enter  it  will  not  be  very  helpful. 
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Figure  8.  Analogy  between  an  elementary  control  system  and 
a  protocol  node.  In  the  control  system,  incoming  perceptual 
data  are  evaluated  and  compared  with  some  reference  that 
represents  the  desired  percept.  The  difference  between  the 
two  results  in  some  virtual  actions  that  are  actually  performed 
by  lower-level  control  systems.  In  the  protocol  node,  the 
information  that  the  partner  should  have  is  compared  with  a 
model  of  what  the  partner  does  have,  the  difference  resulting 
in  a  virtual  message  implemented  by  lower  supporting 
protocols.  The  model  is  affected  by  the  resulting  feedback, 
which  may  result  in  further  virtual  messages. 
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An  error  in  the  command  funtion  is  likely  to  be  more  serious 
and  hard  to  correct  than  an  error  in  a  numeric  parameter, 
because  it  may  be  hard  to  detect,  and  may  cause  unwanted 
actions  to  occur.  It  would  be  unlikely  to  be  corrected  by  the 
pilot  without  a  substantial  period  of  confusion,  annoyance, 
and  expenditure  of  mental  resources.  Accordingly,  there 
might  be  a  need  for  feedback,  if  there  is  a  significant 
probability  that  the  command  itself  might  be  misrecognized 
or  misunderstood.  Alternatively,  the  command  might  be 
entered  tactually,  with  voice  to  provide  the  argument  to  the 
command 

Misrecognition  is  not  the  same  as  misunderstanding.  It  is 
possible  to  understand  a  message  at  a  high  level  while  at  a 
lower  level  misrecognizing  some  of  its  words. 

There  are  two  ways  in  which  the  structure  of  a  high-level 
message  can  be  used  to  increase  the  probability  of  its  being 
correctly  understood:  the  probabilities  for  different  low- 
level  structures  such  as  words  may  be  changed  according  to 
higher-level  expectations  so  that  they  are  likely  to  be  cor¬ 
rectly  recognized  or  misrecognized  words  that  do  not  ft  t  into 
the  high-level  structure  can  be  changed  for  confusable  ones 
that  do  fit. 

The  structure  of  a  high-level  message  involves  everything 
that  the  higher-level  protocol  can  use,  including  the  situational 
context.  For  example,  at  a  particular  stage  in  a  flight,  the  pilot 
may  normally  switch  the  map  display  to  the  more  local 
environment  When  a  flight  reaches  this  stage,  the  probabili¬ 
ties  for  the  appropriate  command  may  be  changed  within  the 
recognizer  itself,  or,  if  the  recognizer  produces  a  nonsensical 
word  string  that  might  easily  have  been  derived  from  a 
sensible  one  that  performs  the  expected  command  the  revi¬ 
sion  might  be  understood  to  have  been  spoken,  as  suggested 
in  Figure  9.  Typically,  in  such  a  situation,  the  higher-level 
protocol  would  determine  that  the  message  was  ambiguous 
or  problematic,  and  would  ask  for  a  check  of  its  putative 


Noise  and  distortion 


Figure  9.  One  way  that  correct  understanding  can  come  from 
misrecognition.  The  mission  state  normallycalls for  achange 
in  the  map  display.  The  pilot  speaks  a  command  for  one,  but 
because  of  noise  and  distortion,  the  recognizer  provides  a 
string  that  would  cause  an  illegal  operation  on  the  radio. 
Because  there  is  a  known  possibility  of  this  kind  of  recognition, 
plus  a  much  higher  probability  for  the  map  command  than  for 
the  illegal  radio  command,  the  high-level  protocol  interprets 
the  command  as  the  one  that  was  really  intended.  Naturally, 
this  interpretation  could  be  checked  by  further  feedback,  such 
as  “Did  you  mean  map  display?” 


understanding,  or  for  a  correction.  This  feedback  might  well 
be  vocal. 

In  order  to  discuss  the  possibilities  for  feedback,  a  digression 
is  needed  on  the  General  Protocol  Grammar  (GPG)  that  we 
believe  underlies  all  dialogue.  Once  the  GPG  is  understood, 
issues  of  how  to  choose  appropriate  feedback  may  become 
clearer. 

The  General  Protocol  Grammar 

Many  authors  argue  that  the  idea  of  a  grammar  for  dialogue 
is  absurd,  and  that  there  cannot  be  such  a  thing,  because 
dialogue  is  so  varied  in  different  situations  (e.g.  Good,  1989). 
We  agree,  but  we  argue  that  there  can  nevertheless  be  a 
grammar  that  describes  the  interactions  within  a  protocol.  In 
fact,  we  argue  that  every  protocol  incorporates  such  a  gram¬ 
mar,  and  that  it  is  the  same  grammar  for  every  protocol.  For 
this  reason,  we  call  it  the  General  Protocol  Grammar  (GPG). 

We  will  describe  the  GPG  in  two  stages.  First,  we  treat  it  as 
a  standard  node-and-arc  state  transition  grammar,  as  shown 
in  Figure  10,  but  only  for  expository  convenience  and  mne¬ 
monic  assistance.  A  state  transition  grammar  works  under 
the  assumption  that  a  state  is  occupied  until  a  discrete 
transition  to  a  new  state  is  made,  and  then  that  new  state  is 
occupied.  In  dialogue,  what  matters  is  information  transmis¬ 
sion,  which  does  not  happen  instantaneously. 

In  the  more  exact  approach  to  the  GPG,  the  focus  is  on  the 
effect  of  information  transmission  on  various  belief  states. 
Changes  in  these  belief  states  correspond  to  state  transitions 
in  the  node-and-arc  grammar. 

The  term  “belief’  should  be  taken  broadly.  Of  course,  the 
aircraft  has  no  beliefs,  and  makes  no  overt  deductions.  It  just 
behaves.  The  aircraft  designer  has  built  the  contingencies  in, 
and  the  aircraft  has  no  possibility  of  changing  them.  In  the 
Layered  Protocol  theory,  this  kind  of  “built-in”  behavior  is  a 
property  of  the  Coders  and  Decoders,  whereas  beliefs  and 
deductions  are  among  the  properties  of  the  Models.  An 
outside  observer,  however,  can  rarely  tell  the  difference.  All 
the  same,  it  seems  reasonable  to  work  as  if  the  lower 
protocols  tend  more  to  “just  behave,”  while  the  higher  ones 
are  more  susceptible  to  analytic  modelling  that  justifies  the 
terms  “believe”  and  “deduce.”  In  both  cases,  we  can  use  the 
same  symbolism  for  the  analysis. 

GPG  as  a  state  transition  network 

A  message  is  initiated  within  a  protocol  when  the  originator 
(O)  has  a  chunk  of  information  that  the  recipient  (R)  should 
have.  The  chunk  may  be  a  part  of  a  higher-level  message 
supported  by  this  protocol,  or  it  may  result  from  some  task 
goal  on  the  part  of  O.  In  either  case,  we  call  it  the  “primal 
message”  that  the  protocol  is  being  asked  to  transmit  It  is  the 
job  of  the  transmitting  protocol  node  to  determine  how  to 
transmit  it,  given  the  beliefs  it  has  about  the  current  situation, 
the  history  of  the  dialogue,  the  recipient’s  beliefs,  and  so 
forth. 

At  one  extreme,  O  (meaning  O’s  transmitting  protocol  node) 
may  believe  that  R  (the  corresponding  receiving  protocol 
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Figure  10.  A  sketch  of  the  General  Protocol  Grammar,  showing  the  major  arcs. 
Circles  represent  stages  at  which  the  originator  of  the  primary  message  must  do 
something,  squares  stages  at  which  the  recipient  of  the  primary  must  do  something. 


node)  already  has  the  information,  and 
will  therefore  transmit  nothing.  For  ex¬ 
ample,  if  the  goal  is  that  the  plane  shou  Id 
fly  straight  and  level,  and  it  already 
appears  to  be  doing  so,  then  the  “joy¬ 
stick  protocol”  should  do  nothing. 

At  the  other  extreme,  O  may  believe  R 
to  know  none  of  the  message.  This  might 
be  the  situation  associated  with  the  en¬ 
try  of  a  flight  plan  into  a  mission  data¬ 
base.  All  the  information  might  have  to 
be  installed  explicitly.  As  an  intermedi¬ 
ate  case,  the  mission  plan  might  be  of  a 
kind  partly  known  to  the  aircraft,  with 
only  someelementstobeexplicitly  trans¬ 
mitted,  such  as  waypoints  or  regions  of 
potential  threat  such  as  SAM  sites. 

In  the  initial  state  of  the  grammar,  O  has  some  belief  about 
what  information  actually  needs  to  be  transmitted,  but  how 
it  is  to  be  transmitted  may  be  a  matter  for  choice,  especially 
at  higher  protocol  levels.  Perhaps  O  attempts  to  encode  the 
whole  message  and  transmits  the  encoding  all  at  once; 
perhaps  O  transmits  only  enough  that  R  is  alerted  to  the  fact 
that  a  message  is  being  sent.  In  either  case,  what  is  sent  is  the 
“Primary”  in  Figure  10,  and  on  receiving  itRmust  determine 
whether  it  could  represent  the  entire  primal  message.  If  it 
could,  then  R  provides  whatever  feedback  is  necessary  to  get 
O  to  move  to  the  node  marked  “Is  it  what  I  want.”  If  not,  R 
provides  feedback  that  indicates  to  O  that  there  is  a  problem. 

Consider  just  these  two  possibilities  in  the  case  in  which  R’ s 
part  of  the  dialogue  is  taken  by  an  isolated  word  speech 
recognizer  running  with  a  wide  open  syntax.  O  speaks  a 
word,  and  R,  the  recognizer,  produces  both  a  best  candidate 
recognition  and  a  “goodness”  measure.  If  the  recognition  has 
a  high  goodness  measure,  then  R  should  “want”  O  to  move 
to  the  “Is  it  what  I  want”  node  at  which  O  might  accept  the 
recognition  as  correct  Otherwise,  R  would  want  O  to  move 
to  the  “Problem”  node  at  which  O  could  solve  the  problem. 
Clearly,  these  two  situations  demand  distinct  feedback  so 
that  O  can  determine  which  possibility  R  intends.  But  most 
recognizers  that  provide  word-by-word  feedback  do  not 
indicate  any  difference  between  high-probability  and  low- 
probability  recognitions.  O  can  find  the  problematic  ones 
only  by  monitoring  each  word  fed  back.  If  the  errors  are  few, 
as  they  often  are  for  good  recognizers,  0  is  likely  not  to  detect 
them.  But  almost  certainly,  errors  will  be  more  probable  for 
the  recognitions  given  a  low  goodness  measure  by  the 
recognizer  than  for  those  given  a  high  goodness  measure. 
There  may  be  occasional  errors  even  in  the  recognitions 
“believed”  by  the  recognizer  to  be  good,  but  it  will  be  hard 
for  the  user  to  detea  them  among  all  the  good  ones.  They  are 
probably  best  passed  through  to  a  higher  protocol  for  detec¬ 
tion  and  correction. 

This  analysis  suggests  the  reason  for  a  practice  that  is 
commonly  found  useful:  to  feed  back  only  those  words  with 
a  low  goodness  measure,  and  to  accept  the  rest  silently, 
passing  them  up  to  a  higher  level  as  if  they  were  correct  O 
can  then  readily  detect  R’s  use  of  the  “Problem”  arc  in  the 
grammar,  and  can  make  appropriate  corrective  or  accepting 


responses.  An  additional  benefit  is  that  R  should  not  expect 
any  response  to  the  well  recognized  words,  but  should  expect 
a  confirmatory  or  corrective  message  about  each  potential 
misrecognition.  O  is  thereby  relieved  of  the  need  to  identify 
for  R  which  word  is  being  corrected,  as  O  would  have  to  do 
if  R  reported  each  word.  There  is  a  double  benefit  for  the 
resource  loading  on  O:  the  monitoring  of  feedback  for 
incorrea  words  is  eased,  and  O  has  less  to  do  to  correct  a  false 
recognition. 

In  the  GPG,  the  different  arcs  may  be  instantiated  in  different 
ways,  depending  on  what  the  partners  believe  each  other  to 
believe.  For  example,  suppose  that  R  (in  any  protocol,  not 
just  word  recognition)  believes  that  the  primary  message  has 
provided  enough  information  to  permit  a  satisfactory  inter¬ 
pretation  of  the  primal  message.  R  wishes  O  to  move  to  the 
node  “Is  it  what  I  want”  in  Figure  10.  IfR  believes  thatO  will 
be  sure  that  R  did  get  a  good  message  and  would  interpret  it 
properly,  then  no  overt  feedback  is  necessary.  We  call  this  a 
“null  instantiation”  of  the  Normal  Feedback  arc.  A  second 
possibility  is  that  R  believes  that  O  does  not  know  whether 
R  has  made  a  good  interpretation,  but  would  trust  R’s 
judgment  in  the  matter.  All  R  need  do  is  to  indicate  that  the 
message  was  received.  We  call  this  a  “neutral  instantiation” 
of  the  Normal  Feedback  arc.  There  are  at  least  two  other 
instantiations  for  this  arc,  “informative”  and  “corrective”; 
most  arcs  in  the  GPG  have  more  than  one  instantiation. 

The  state  transition  description  of  the  GPG  is  in  principle 
adequate  for  many  purposes.  In  its  full  form,  there  are  many 
more  arcs  than  are  shown  in  Figure  10 — about  24  arcs  in  one 
recent  version,  with  an  average  of  about  two  instantiations 
for  each  arc.  But  a  state  transition  diagram  is  inadequate  for 
real  communication,  in  which  partial  interpretations  are 
continuously  being  made  at  all  protocol  levels.  To  handle  this 
situation,  we  must  consider  the  models  and  beliefs  that 
underlie  the  state  transition  description  of  the  GPG. 

Belief  structure  of  the  GPG 

The  GPG  is  a  description  of  the  process  of  communicating  a 
virtual  message.  It  is  not  concerned  with  the  content  or  level 
of  abstraction  of  the  message.  Accordingly,  the  information 
that  directs  the  state  transitions  is  concerned  not  with  the 
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message  itself,  but  with  the  satisfaction  of  the  partners  about 
its  transmission. 

We  identify  three  propositions  dealing  with  the  message 
transmission: 

•  PI:  R  has  made  an  interpretation  of  the  message. 

•  P2:  If  PI,  then  R’s  interpretation  is  satisfactory. 

•  P3:  it  is  not  worthwhile  to  continue  trying  to  transmit 

the  message. 

These  three  propositions  are  statements  of  fact  Facts  are 
unknowable,  butanyone  is  entitled  to  believe  them  with  any 
degree  of  certainty  from  strong  disbelief  through  total  igno¬ 
rance  to  implicit  faith.  We  assign  the  degree  of  belief  that 
someone  hais  about  one  of  these  facts  a  number  from  - 1  to + 1 . 
We  notate  the  degree  of  belief  A  holds  about  P  as  A(b,P),  or 
simply  A(P),  omitting  the  b  because  by  default  we  are 
dealing  with  belief.  On  other  occasions  we  use  a  similar 
notion  for  the  strengths  of  goals  and  intentions,  using  g  or  i, 
which  are  ordinarily  not  omitted  in  the  notation. 

The  final  element  of  notation  is  to  divide  the  degrees  of  belief 
into  five  categories,  as  a  matter  of  convenience  rather  for  any 
reason  of  theory:  Disbelief  (D),  Negative  belief  (N),  igno¬ 
rance  (X),  weak  belief  (W),  and  Strong  belief  (S).  The  letters 
correspond  to  numerical  ranges  that  are  not  well  specified. 
The  boundaries  between  the  ranges  occur  at  points  where  the 
behaviour  of  one  or  other  partner  may  change. 

Given  this  notation,  we  can  write  statements  such  as  W<  A(P), 
which,  by  convention,  means  that  A  holds  at  least  a  weak 
belief  in  P,  or  S=A(b,W<B(P)>,  which  means  that  A  strongly 
believes  that  B  at  least  weakly  believes  P.  S=  A(g,W<B(P» 
means  that  A  holds  a  strong  goal  that  B  should  at  least  weakly 
believe  P. 

Now  consider  the  three  propositions  of  the  GPG.  So  long  as 
a  message  is  still  being  transmitted,  each  partner  must  at  least 
weakly  believe  that  P3  is  false: 

•  W<0{b,-^>3)  &  W<R(b,->P3) 

Implicit  in  this  is  that  at  least  one  of  the  partners  believes  that 
either  PI  or  P2  is  false  and  that  each  has  a  goal  that  both  PI 
and  P2  should  become  true. 

This  notation  may  seem  a  little  removed  from  the  problem  of 
flying  an  aircraft,  but  let  us  apply  it  to  the  simple  matter  of 
adjusting  the  climb  angle,  using  the  joystick.  We  shall  call 
the  pilot  C  (commander)  and  the  aircraft  A. 

The  design  of  the  aircraft  suggests  that  both  A  and  C 
“believe”  that  P2  is  always  true  (in  the  absence  of  malfunc¬ 
tion).  In  other  words,  if  the  pilot  manipulates  the  stick 
appropriately,  the  aircraft  will  assume  the  correct  attitude. 
But  the  variety  of  flight  situations  assure  that  simply  setting 
the  joystick  to  some  condition  for  a  fixed  duration  will  not 
work.  The  pilot  must  receive  feedback  from  the  aircraft  as  to 
its  present  (and  perhaps  predicted)  attitude.  This  is  normally 
done  through  an  instrumentdisplay  in  current  aircraft,  though 
in  earlier  days  the  pilot  had  to  look  outside.  We  can  take  the 
instrument  display  as  feedback  for  the  joystick  message. 

In  the  state  transition  diagram  of  the  GPG,  the  behavior  is 
only  crudely  described.  The  pilot  makes  an  initial  adjustment 
(“primary”),  the  plane  assumes  some  attitude  (“Normal 
Feedback”),  the  pilot  determines  whether  it  is  the  desired 


attitude,  and  if  not,  uses  the  “Edit”  arc  followed  by  “Ac¬ 
cepted  Resolution”  as  the  plane  assumes  a  new  attitude.  The 
“Edit”  loop  is  continued  until  the  attitude  is  correct. 

Such  discrete  control  is  a  very  poor  way  to  handle  an  aircraft. 
Effective  control  is  continuous,  the  pilot  judging  the  attitude 
and  the  rate  of  change  of  attitude,  and  manipulating  the 
joystick  continuously  to  achieve  as  quickly  as  possible  the 
perception  that  the  attitude  is  correct  The  protocol  feedback 
loop  and  the  types  of  message  involved  are  shown  in  Figure 
11. 

In  terms  of  the  belief  structures,  the  pilot  starts  with  a  strong 
belief  that  the  aircraft  has  not  interpreted  his  intention  that  it 
take  some  particular  attitude 

•  S=C(-.P1) 

The  aircraft  has  no  belief,  as  the  word  is  ordinarily  used  in 
English,  since  models  are  not  built-in  to  most  aircraft,  but  a 
future  computer-controlled  aircraft  might  well  develop  some 
beliefs.  In  fact,  one  might  consider  that  the  Mulhouse  A-320 
crash  as  being  due  to  a  belief  held  by  the  aircraft  that  differed 
from  the  intentions  of  the  pilot  about  the  future  path  of  the 
plane.  Whether  or  not  this  is  the  case,  one  can  in  the 
formalism  assert  that  by  design,  if  the  pilot  moves  the 
joystick  the  aircraft  “believes”  that  it  holds  the  wrong  atti¬ 
tude.  Hence,  the  joystick  movement  leads  to: 

•  S=A(P1  &  -iP2) 

It  has  interpreted  (and  acted  upon)  the  message  (joystick 
movement)  but  “believes”  its  interpretation  is  incorrect 
because  the  joystick  continues  to  be  moved. 

As  the  aircraft’s  attitude  approaches  the  desired  one,  the 
pilots’s  belief  structure  changes,  so  that  strong  belief  in  the 
wrongness  of  the  attitude  gives  way  to  weak  belief,  indiffer¬ 
ence,  and  disbelief.  At  that  point,  the  joystick  is  neutralized, 
so  that  the  aircraft  can  now  “believe” 

•  S=A(W<C(P3)) 
and  hence 

•  S=A(W<C(P1  &  P2)). 

The  aircraft  “deduces”  from  the  pilot’s  cessation  of  moving 
the  joystick  that  the  pilot  is  satisfied  with  its  interpretation  of 


Figure  1 1 .  The  protocol  loop  for  the  control  of  aircraft  attitude. 
The  pilot  sends  a  continuous  stream  of  messages  by 
manipulating  the  joystick  and  other  controls,  which  are  sent 
by  means  of  whatevercontrol  mechanisms  the  aircraft  provides 
(cables,  hydraulics,  computer-driven  servos)  to  the 
aerodynamic  control  surfaces.  The  aircraft  signals  the 
interpretation  of  this  message  by  means  of  changes  to  the 
cockpit  displays. 


21-11 


the  message,  which  is  to  say  that  he  is  satisfied  with  its 
current  attitude. 

It  is  important  to  note  that  the  changes  in  belief  are  continu¬ 
ous,  and  occur  in  parallel  with  the  actions  of  both  partners — 
the  pilot  and  the  aircraft.  It  is  this  continuously  developing 
and  parallel  communication  that  cannot  be  handled  by  the 
state  transition  description  of  the  GPG.  Neither  can  it  be 
handled  by  a  turn-taking  analysis  of  dialogue.  It  is,  however, 
characteristic  of  most  realistic  communication,  whether 
among  people  or  between  people  and  their  “intelligent” 
machines. 

Voice  and  belief  structure 

When  we  come  to  consider  voice  interaction,  the  concepts  of 
“belief’  and  “deduction”  become  more  plausible,  since  voice 
recognition  equipment  is  normally  based  on  probabilistic 
models.  The  recognizer  starts  with  some  set  of  expectations 
about  the  possible  things  the  user  might  say.  and  these 
expectations  often  include  probabilities  that  might  well  be 
situation-dependent  In  the  LP  structure,  the  probabilities 
will  depend  also  on  the  momentary  state  of  interpretation  of 
the  higher-level  message  that  it  supports. 

Let  us  suppose  that  the  higher  level  message  is  a  flight  plan 
consisting  of  a  set  of  waypoints  identified  by  name  from  the 
hypothetical  map  in  Figure  12.  Let  us  also  suppose  that  the 
aircraft  is  “intelligent”  in  that  it  has  a  database  of  locations 
and  some  concept  of  what  a  normal  flig at  plan  might  look 
like.  In  particular,  a  normal  flight  plan  does  not  double  back 
on  itself.  Now  the  pilot  starts  to  enter  waypoints:  Astal, 
B  irland,  Demick,  Coltaine,  Endow. 

Suppose  the  pilot  makes  a  mistake  and  enters  Demick  before 
Coltaine.  The  recognizer  correctly  identifies  the  words  from 
the  acoustic  signal.  The  protocol  at  the  next  level  translates 
these  words  into  coordinate  strings  that  will  be  entered  into 
the  navigation  system.  But  the  Model  in  inis  protocol  could 
incorporate  the  knowledge  that  normal  routings  do  not 
include  consecutive  turns  near  180°,  and  this  knowledge 
would  allow  it  to  propose  that  the  message  is  possibly 
incorrect.  In  the  state  transition  grammar,  the  aircraft  would 
use  the  “Problem”  arc  in  Figure  10.  In  the  belief  notation, 

•  W<A(P1  &-P2&  W<C(P1  &  P2)) 
which  indicates  a  difference  between  its  view  of  the  correct¬ 
ness  of  the  message  and  the  view  it  believes  the  pilot  has.  The 
result  should  therefore  be  to  inform  the  pilot  not  only  that  it 
finds  the  message  dubious,  but  also  in  what  way  it  finds  it 
dubious.  Accordingly,  its  feedback  should  be  to  propose  a 
reasonable  reinterpretation:  “Do  you  mean  B  irland,  Coltaine, 
Demick?”  The  pilot,  of  course,  might  have  intended  the 
zigzag  route  that  was  originally  entered,  and  could  say  so,  by 
voice. 

None  of  the  above  indicates  the  manner  in  which  the  feed¬ 
back  is  provided.  There  are  some  criteria  that  could  help  to 
determine  that;  the  method  must  alert  the  pilot  that  the 
feedback  exists,  and  it  must  be  able  to  convey  the  informa¬ 
tion  as  to  what  the  problem  might  be.  It  cannot,  therefore,  be 
provided  by  a  non-obvious  visual  display,  unless  the  prob¬ 
ability  of  error  is  high  enough  that  the  pilot  will  normally 
look  to  see  whether  one  had  occurred.  If  such  is  the  case,  the 


same  display  could  support  feedback  from  both  the  word 
recognizer  protocol  and  the  waypoint  construction  protocol. 

Integration 

The  idea  of  “integration”  is  closely  bound  with  that  of 
diviplexing,  but  the  integration  of  voice  with  non-vocal 
interaction  between  pilot  and  aircraft  usually  does  not  in¬ 
volve  diviplexing  at  any  low  level.  The  voice  usually  re¬ 
places  rather  than  supplements  the  non-vocal  interactions, 
though  it  could  affect  the  interpretation  the  aircraft  places  on 
non-vocal  actions.  For  example,  a  navigation  display  might 
be  presented  on  a  touch-sensitive  screen,  and  the  area  dis¬ 
played  changed  under  voice  control.  Likewise,  normally  the 
non-vocal  interactions  do  not  supplement  the  voice.  Each 
acts  essentially  independently,  except  that  they  both  affect 
the  behavior  of  the  aircraft. 

Consider  the  waypoint  entry  example  illustrated  in  Figure 
12.  Suppose  the  plane  were  on  the  leg  Coltaine-Demick  and 
the  pilot  spoke  a  command  such  as  “Next  waypoint  B  irland.” 
The  flight  situation  would  allow  the  waypoint  protocol 
Model  to  detennine  that  this  would  entail  a  turn  near  180°, 
and  might  suggest  the  possibility  of  a  misinterpretation  that 
should  be  queried.  This  query  would  be  independent  of 
whether  any  waypoints  had  previously  been  entered  by 
voice,  but  would  be  based  on  the  present  situation  of  the 
aircraft 

Another  way  in  which  the  different  modes  of  interaction 
relate  to  each  other  is  that  each  imposes  a  load  on  the  pilot’ s 
resources.  The  pilot  must  be  kept  aware  of  the  local  situation, 
and  alerted  to  possible  dangers  and  opportunities,  but  to 
provide  too  great  a  flood  of  information  would  risk  the  pilot 
ignoring  important  events.  The  problem  of  evaluating  situa¬ 
tion  awareness  and  workload  is  a  long-standing  one  in 
psychology,  and  no  solution  is  near.  Nevertheless,  qualita¬ 
tive  analyses  could  easily  assist  the  interface  designer  in 
determining  how  and  when  to  provide  feedback  or  to  request 
pilot  action. 

If  a  particular  interaction  would  benefit  from  joint  use  of  the 
symbolic  information  transmitted  by  voice  and  the  continu¬ 
ous  information  transmitted  by  control  movements,  the  struc¬ 
ture  of  the  messages  that  immediately  support  the  high-level 
message  must  contain  the  information  that  allows  them  lobe 

Endow 


Figure  1 2.  A  hypothetical  map  with  some  named  waypoints. 
The  natural  route  from  Startow  to  Endow  follows  the  solid 
arrows.  Erroneously  ordered  input  could  lead  to  the  route 
shown  by  dashed  arrows. 
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recombined.  For  example,  a  map  display  contains  labelled 
entities  that  could  well  have  descriptive  information  avail¬ 
able  in  adatabase.  The  geographic  spatial  information  is  well 
displayed  pictorially,  and  the  items  can  be  selected  verbally, 
but  the  notion  of  “SAM  sites  north-east  of  Demick”  involves 
a  linkage  between  the  verbal  and  the  visual,  through  the 
coordinate  location  of  Demick,  which  is  at  once  symbolic 
and  selected  from  a  continuum  of  possible  locations. 

CONCLUSION 

A  basic  principle  of  behavior  is  that  purposive  actions  can  be 
performed  only  with  respect  to  some  percept  that  the  actions 
will  affect,  based  on  sensory  data.  A  natural  corollary,  which 
leads  to  Perceptual  Control  Theory  (PCT),  is  that  actions  are 
normally  such  as  to  make  the  percept  closer  to  some  desired 
state,  or  reference  percept.  Behavior  thus  controls  percep¬ 
tion. 

PCT  contains  the  concept  of  an  Elementary  Control  System, 
a  structure  that  accepts  sensory  input  from  a  variety  of 
sources,  combines  them  into  a  percept  through  some  percep¬ 
tual  function,  compares  the  percept  with  a  reference  to  create 
an  error  signal,  and  distributes  some  function  of  the  error 
signal  to  effectors.  Both  the  sources  of  sensory  data  and  the 
effectors  may  well  be  themselves  Elementary  Control  Sys¬ 
tems  at  a  lower  level  in  a  hierarchy,  so  that  the  reference 
signal  in  any  Elementary  Control  System  is  a  combination  of 
the  error  signals  from  higher  ones.  The  perceptual  control 
system  as  a  whole  consists  of  many  layers  of  Elementary 
Control  Systems,  the  lowest  ones  acting  directly  on,  or 
receiving  data  directly  from,  the  physical  world. 

The  theory  of  Layered  Protocols  asserts  that  a  similar  struc¬ 
ture  describes  the  interactions  among  “intelligent”  entities, 
which  are  entities  that  have  three  independences:  independ¬ 
ence  of  design,  independence  of  sensing  mechanism,  and 
independence  of  action.  Messages  are  passed  by  means  of 
creating  effects  in  the  physical  world,  but  the  physical 
messages  represent  messages  of  a  hierarchy  of  levels  of 
abstraction.  The  passage  of  a  message  at  any  level  of  abstrac¬ 
tion  uses  a  specific  protocol,  which  includes  a  general 
protocol  grammar  (GPG)  that  determines  the  kinds  of  feed¬ 
back  that  are  appropriate  to  different  conditions  in  the 
sending  of  the  message. 

The  GPG  can  be  applied  to  the  manual  control  of  an  aircraft, 
as  well  as  to  both  simple  and  complex  voice  interaction.  The 
problem  of  integrating  voice  with  other  control  mechanisms 
thus  becomes  partly  one  of  determining  which  kinds  of 
message  suit  the  symbolist  character  of  voice,  and  which  suit 
the  continuous  character  of  manual  control.  In  aircraft,  the 
second  issue  of  integration  is  less  important:  how  to  ensure 
that  a  high-level  message  transmitted  by  voice  and  manual 
control  is  properly  reintegrated  by  the  aircraft.  Mission 
situation  can  be  effective  in  determining  the  interpretation  of 
high-level  messages  supported  in  part  by  voice,  even  when 
the  recognizer  correctly  reports  words  erroneously  spoken 
by  the  pilot. 

Layered  Protocols  provides  a  principled  framework  for 
describing  and  integrating  messages  of  di  fferent  kinds,  espe 
daily  in  complex  interfaces  such  as  the  aircraft  cockpit. 


NOTE  ADDED  AFTER  THE  SYMPOSIUM: 

Graceful  control  of  automated  systems 

On  the  ilrst  day  of  the  workshop,  several  talkers  made  the 
point  that  pilots  had  a  difficult  time  accepting  automated 
functions  beyond  the  most  simple,  although  they  indicated  in 
questionnaires  that  they  wanted  them.  What  the  pilots  did  not 
want  was  for  the  automated  functions  to  take  decisions  at 
critical  moments  that  they  would  rather  take  for  themselves, 
although  the  automated  function  could  perform  non-critical 
duties.  This  problem  seems  to  be  readily  addressed  within 
the  PCT  framework. 

Both  the  plane  and  the  pilot  are  conceived  as  hierarchic 
control  systems,  the  plane's  upper-level  references  being  set 
either  by  the  designer  or  by  the  pilot.  For  examples,  the  pilot 
gives  the  autopilot  a  reference  to  keep  the  plane  on  a  certain 
beading  at  a  certain  altitude  and  with  a  stable  attitude 
regardless  of  winds.  The  pilot  resets  this  reference  from  time 
to  time,  or  she  might  set  a  whole  sequence  of  waypoints  from 
which  the  plane  computes  references  for  the  autopilot.  In 
either  case,  if  we  think  of  the  pilot  and  plane  as  one  single 
hierarchic  control  system,  the  plane’s  chunk  simply  takes 
over  the  function  of  performing  the  task  of  satisfying  the 
references  provided  by  the  pilot. 

If  the  pilot  has  delegated  control,  the  plane  taking  over  the 
particular  function,  the  pilot  tends  to  lose  "situation  aware¬ 
ness”  in  respect  of  that  function.  He  could  control  the 
function  if  he  wanted  to,  but  since  he  is  not,  neither  is  he 
acquiring  the  sensory  information  that  would  allow  him  to 
get  the  perception  that  he  could  be  controlling.  He  may  not 
perceive  what  is  going  on.  The  pilot's  re-acquisition  of 
situation  awareness  when  retaking  control  of  an  automated 
function  is  a  significant  problem.  To  continue  the  autopilot 
example,  the  autopilot  is  switched  outof  the  control  loop,  and 
the  pilot’s  own  lower-level  control  systems  take  over  the 
maintenance  of  heading,  altitude,  and  attitude.  One  signifi¬ 
cant  case  in  which  this  might  happen  is  collision  avoidance, 
where  situation  awareness  is  critical. 

The  view  of  the  automated  function  being  switched  in  or  out 
of  the  loop  in  alternation  with  the  equivalent  part  of  the  pilot's 
hierarchy,  as  in  Figure  1 3,  is  almost  inevitable  with  convcn- 


Figure  13.  Conventional  switched  mode  of  operation  of 
automated  functions.  When  the  aircraft  is  controlling  3 
function,  the  pilot  is  not. 
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tional  approaches  to  the  problem.  But  PCT  offers  a  different 
solution.  Imagine  that  instead  of  a  simple  switch  that  sends 
a  reference  signal  either  to  part  of  the  plane's  hierarchy  or  to 
part  of  the  pilot's,  the  reference  signal  is  sent  always  to  both, 
asin  Figure  14.  If  the  pilot  is  choosing  not  to  control,  the  gain 
in  her  part  of  the  loop  is  zero.  The  gain  in  the  aircraft's  part 
of  the  loop  is  adequate  to  maintain  course  against  external 
disturbances. 

But  it  is  possible  for  the  pilot  to  set  his  gain  to  some  low  value 
other  than  zero,  and  "shadow"  the  aircraft's  control.  The 
aircraft  could  sense  this  in  two  ways.  One  way  is  that  the 
pilot's  attempts  to  control  might  set  up  aconflict  in  the  lower- 
level  systems  that  actually  drive  the  plane's  control  surfaces. 
The  result  would  be  a  persistent  failure  of  the  automated 
system  to  achieve  the  desired  percept,  if  the  pilot's  references 
differed  from  those  of  the  plane.  The  second  is  that  in 
contrast  to  ordinary  disturbances,  the  pilot's  actions  can  be 
directly  sensed  by  the  plane,  as  shown  by  the  dashed  arrow 
in  Figure  14.  So  long  as  the  pilot's  gain  remains  low,  the 
automated  system  would  keep  its  own  gain  high,  but  as  soon 
as  the  pilot's  gain  increased,  the  plane  would  drop  the  gain  of 
the  automated  system,  perhaps  to  zero.  The  pilot  is,  at  low 
gain,  maintaining  situation  awareness,  or  regaining  it  pre¬ 
paratory  to  taking  control. 

There  is  a  continuum,  as  the  plane's  gain  decreases,  between 
the  plane  performing  the  function,  assisting  (and  perhaps 
training)  the  pilot  to  perform  it,  and  getting  out  of  the  way  to 
let  the  pilot  do  what  she  wants.  There  is  no  need  for  the  pilot 
to  switch  automated  functions  in  and  out;  they  are  in  by 
default,  but  as  soon  as  the  pilot  starts  controlling  what  they 
control,  they  gracefully  get  out  of  the  way. 

What  the  pilot  can  switch  in  or  out,  or  alter  in  a  continuous 
way,  is  the  sensitivity  of  the  plane  to  the  pilot's  insistence  on 
control.  A  novice  pilot  could  set  a  high  level,  asking  the 


Figure  14.  A  PCT-based  view  of  conjoint  manual  and 
automated  control.  The  automated  system  maintains  control 
until  it  senses  that  the  pilot  is  controlling  the  “same”  percept, 
when  it  reduces  its  gain  or  even  stops  entirely,  until  the  pilot 
aw^mpting  to  control  that  percept.  At  low  gain,  the 
automated  system  can  assist  or  train  a  novice  pilot. 


plane  to  do  what  it  thinks  proper  even  though  she  requires  it 
moderately  strongly  to  do  something  else,  whereas  an  expert 
would  want  it  to  get  out  of  the  way  as  soon  as  she  started 
controlling. 

Shifts  of  control  locus  need  affect  only  a  small  part  of  the 
control  hierarchy.  The  pilot's  choice  to  control  the  course  of 
the  plane  does  not  indicate  that  he  must  control  the  positions 
of  individual  control  surfaces.  And  the  plane  can  know  at 
what  level  the  pilot  does  desire  to  take  control.  The  pilot 
may,  for  example,  take  quick  evasive  action  (controlling  the 
momentary  attitude  and  course  of  the  plane)  while  leaving 
the  automated  systems  in  charge  of  the  control  surfaces  and 
the  attainment  of  the  waypoints  that  define  the  larger  course 
of  the  plane. 
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NOTE 

This  paper  owes  much  to  discussions  over  several  years 
within  NATO  AC243  (Defence  Research  Group)  Panel  3 
Research  Study  Group  lOon  Automatic  Speech  Processing, 
and  to  electronic  interactions  with  W.  T.  Powers  and  other 
participants  in  the  Internet  mailing  list  CSG-L. 
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Discussion 


QUESTION  R.M.  TAYLOR 

Your  description  of  the  application  of  perceptual  control  theory  to  the 
control  of  the  aircraft's  state  seems  clear  and  useful.  However,  the 
designer's  main  task  in  the  future  will  be  to  help  the  pilot  to  solve 
mission  problems.  Can  you  forsee  any  changes  to  perceptual  control  theory 
that  will  be  necessary  in  order  to  apply  it  to  solving  mission  tactical 
problems  and  pilot  tactical  decision-making? 

REPLY 

Perceptual  Control  Theory  (Powers,  1973)  has  many  different  kinds  of 
abstraction  -  eleven  at  the  current  state  of  the  theory,  including 
categories,  sequences,  programs,  principles,  among  others.  Mission 
problems  seem  related  to  programs  and  sequences,  which  are  components  of 
current  theory,  so  to  that  extent  no  changes  are  necessary.  On  the  other 
hand,  the  precise  behaviour  of  non-linear  control  systems  is  not  well 
understood  and  neither  is  the  behaviour  of  neural  network  meshes,  which 
form  a  part  of  the  control  network.  Hence  the  theory  is  likely  to  provide 
a  framework  for  productive  analysis  and  design  rather  than  a  precise 
predictive  model.  The  key  point  is  to  realise  that  the  stabilities  of 
behaviour  relate  to  the  intentions  of  the  actor  and  not  to  the  actions 
taken  to  realize  those  intentions. 
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ABSTRACT 

The  aim  of  this  study  was  to  find, 
from  the  computer  signal  analysis,  acoustic 
features  of  the  speech  production  with  G-load 
effects.  Their  correlates  with  physiological 
features  wilt  permit  a  interpretation  of  the 
variability  of  formant  and  pitch  shifts. 

The  pilot  study  experiments  were 
conducted  by  SEXTANT  and  the  LAMAS 
(Laboratoire  de  Medecine  a6rospatiale)  du 
centre  d'essais  en  vol  de  Bretigny.  Six 
subjets  participated  in  the  experiment.  Their 
mean  age  was  30  years.  A  series  of  experi¬ 
ments  in  centrifuge  environment  have  been 
performed  as  part  of  a  research  program. 

A  specific  vocabulary  has  been 
made  for  the  first  investigation  with 
speakers  in  centrifuge.  Comparison  of 
spectrographic  analysis  and  wavelet 
decomposition  have  permited  to  show  a 
spectral  pattern  modification  for  recognition 
process. 

Oue  to  the  accuracy  and  limi¬ 
tations  of  the  sono-graph  measurements,  we 
have  developed  multiresolution  transforms.  A 
window  Fourier  transform  is  better  suited  for 
analysing  signals  where  all  patterns  appear 
approximately  at  the  same  scale. 

The  old  multivariate  statistical 
analysis,  after  the  Bark's  transformation, 
can  produce  a  good  projection  on  the 
eigenvectors  of  the  correlation  matrix.  The 
choice  of  the  eigenvalues  seems  very  easy  for 
seeking  the  best  representation  of  speech 
production. 

An  interpretation  of  the  mecha¬ 
nical  effects  due  to  the  acceleration  can 
been  shown  with  analysis  tools.  We  observed, 
in  agreement  with  acoustic  multi  resolution 
analyses,  that  producing  speech  in  G-load 
conditions  can  increase  energy,  pitch  and 
formant  frequency  locations.  The  influence  of 
the  breathing  on  some  parameters  is 
noticeable. 

The  last  section  of  the  paper 
discusses  the  incidence  of  these  results  on 
the  robust  speech  recognizer  for  military 
cockpit  environment. 


RESUME 

Le  but  de  ce  travail  6tait  de 
d£crire  les  modifications  de  I'eiocution  sous 
facteur  de  charge  6  partir  d'outils 
d'analyse.  Leurs  correlations  avec  des  effets 
physiologiques  permettent  de  fournir  une 
interpretation  de  la  variability  formantique 
et  du  d6cal age  de  la  frequence  fondamentale 
des  voisements. 

Une  experimentation  a  6te  men6e 
conjointement  entre  SEXTANT  et  le  LAMAS 
(Laboratoire  de  Medecine  Aerospatiale).  Six 
sujets  d'age  moyen  30  ans  ont  particip6  aux 
experimentations  en  centrifugeuse. 

Un  vocabulaire  specifique  deva¬ 
luation  a  6te  mis  au  point,  des  tests  ont 
compare  les  sonogrammes  classiques  aux  decom¬ 
positions  sur  des  bases  d'ondelettes  orthogo¬ 
nal  es  et  continues. 

En  raison  de  la  precision  et  des 
limitations  inh6rentes  au  sonogramme,  nous 
,-vons  developpe  des  transformations  multi - 
resolution.  Le  choix  de  I’analyse  de  Fourier 
se  reveie  en  effet  judicieux  uniquement 
lorsque  les  phenom6nes  i  analyser  ont  la  meme 
echelle  et  s ont  e  structure  harmonique. 

L'analyse  multi -variable  en 
composantes  principales  apr6s  transformation 
de  Bark  a  permis  de  mettre  en  evidence  des 
disparites  en  comparant  les  projections  sur 
plusieurs  vecteurs  propres  de  la  matrice  des 
correlations.  Le  choix  des  valeurs  propres 
semble  ais6  et  concourt  a  la  visualisation 
exhaustive  de  la  production  vocale. 

Une  interpretation  des  effets 
mecaniques  lies  d  l 'acceleration  a  6t6  pos¬ 
sible  grace  aux  outils  mis  en  place.  Nous 
avons  observe  conform6ment  &  l'analyse  multi- 
resolution  une  -ugnentation  notable  de 
I'energie,  de  la  frequence  du  fondamental  et 
de  la  variability  formantique.  L' influence  de 
la  respiration  fait  partie  des  indices 
penalisant  la  prosodie  done  la  reconnais¬ 
sance. 

La  dernifere  section  traite  de 
('incidence  de  ces  resultats  sur  le 
durcissement  d'un  systeme  de  reconnaissance 
pour  avion  d'armes. 
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I)  Conditions  envi  romementa  les 

En  reconnaissance  automatique  de  la  pa¬ 
role,  les  taux  de  reconnaissance  obtenus  pour 
des  applications  "laboratoire"  ne  sont  pas 
directement  transposables  aux  envi ronnements 
difficiles,  type  avion  d'armes.  La  chute  des 
scores  de  reconnnaissance  s'explique,  non 
seulement  par  la  presence  de  bruit,  mais  6ga- 
lement  par  la  variability  acoustique  induite 
par  le  locuteur  pour  s' adapter  aux  contraintes 
envi  romementa  les. 

L'amyi ioration  des  performances  des 
systymes  de  reconnaissance  en  conditions 
rye l les  passe  par  une  meilleure  comprehension 
des  facteurs  dimmensionnants;  en  raison  de  la 
multiplicity  des  conditions  d'utilisation  des 
systymes  opyrationnels  la  description 
exhaustive  des  diffyrents  paramytres 
const itue  une  lourde  tache. 

Nous  avons  voulu  au  cours  de  ce  tra¬ 
vail,  ytudier  les  incidences  du  facteur  de 
charge  pris  isolement  sur  la  production  de 
parole  et  par  deiy  sur  le  systyme  de  recon¬ 
naissance  en  vue  de  dygager  un  durcissement  de 
l 'algorithmic  du  systyme  global. 

Le  panorama  des  travaux  realises  a  ce 
jour  sur  la  parole  sous  facteur  de  charge 
parait  restreint;  nous  dycrivons  les  effets  de 
l 'accyiyration  sur  les  paramytres  acoustiques 
de  I'yiocution.  Afin  de  mieux  analyser  les 
rysultats  de  la  campagne  d'essais  nous  avons 
dyveloppy  une  syrie  d'outils  discriminants. 

Nous  yiaborons  par  la  suite  une 
mythodologie  expyrimentale  pour  mettre  en 
yvidence  l' influence  du  facteur  de  charge  sur 
la  production  de  parole.  Nous  donnons  pour 
finir  une  interprytation  des  effets 
mycaniques.  Nous  montrons  l 'utility  d'une 
detection  automatique  de  parole  fiable  et 
robuste  associye  au  systyme  de  reconnaissance 
proposy. 

La  communication  verbale  s'articule 
suivant  di verses  ytapes  qui  relient  le 
concepteur  du  message  au  cerveau  de 
I'auditeur.  Le  premier  est  de  niveau  linguis- 
tique  il  correspond  aux  choix  des  di f f y- 
rentes  unitys  signifiantes  par  le  locuteur; 
le  niveau  suivant  est  articulatoire,  le 
processus  d'encodage  se  termine  par  la 
production  d'une  onde  sonore. 

La  nature  trys  particulidre  des 
enregistrements  ryalisys  pendant  des  dycen- 
nies  dans  des  conditions  dites  de 
"laboratoire"  a  conduit  un  grand  nombre  de 
spycialistes  d  proposer  une  interprytation 
incompiyte  de  la  variability  acoustique  et  y 
rechercher  des  invariants. 

En  effet  pour  certains  auteurs,  la 
variability  acoustique  est  prysentye  comme  la 
rysultante  passive  des  diffyrences 
morphologiques  entre  individus;  elle  est 
considyrye  hors  de  l 'organisation 
linguistique. 

Si  cette  interprytation  peut  s'appli- 
quer  A  l 'analyse  de  la  parole  classique,  elle 
ne  parvient  nullement  A  expliquer  les  modifi¬ 
cations  majeures  qui  apparaisent  pour  I'yio¬ 
cution  en  ambiance  syvyre  :  pour  un  meme 
individu  la  valeur  formantique  est  fortement 
versatile  et  tributaire  de  I'effet  Lombard. 


Ces  ycarts  acoustiques  ne  sont  inter- 
prytables  que  si  l 'on  admet  qu'ils  rysultent 
de  facteurs  actifs  l  i  6s  aux  stratygies 
d' adaptation  et/ou  de  compensations 
articulatoires  mises  en  oeuvre  en  situation 
ryelle. 

Les  pilotes,  pour  augmenter  leur 
toiyrance  au  facteur  de  charge  effectuent 
diffyrentes  manoeuvres  de  tension  (appeiyes 
manoeuvres  anti-G).  Ces  efforts  de  tension 
muscutaire  et  d'expi ration  forcye  rysultent 
de  la  combinaison  de  trois  exercices 
spycifiques  ; 

-  pousser  la  tete  vers  le  bas  entre  les 
ypaules  pour  ryduire  la  distance  coeur-oeil, 

tendre  au  maximun  les  muscles 
abdominaux, 

-  augmenter  la  pression  intrathoracique 
par  la  mythode  d'expi ration  forcye.  Ces 
manoeuvres  modifient  notablement  le  pattern 
respiratoire  [2], 

La  variability  acoustique  porteuse 
d' informations  linguistiques  a  pour  origine  : 

-  des  facteurs  inter- individuels, 

-  des  indices  intra- individuels, 

-  I 'instability  de  I'environnement, 

-  I 'omni pr6 sene e  de  fluctuations  aiya- 

toires. 

Les  capacitys  de  dycodage  de  l 'Homme 
intygrent  ces  sources  de  disparitys  qui,  loin 
de  const ituer  une  gene  permettent,  au 
contraire,  de  maintenir  l' intelligibility  du 
message  vocal  en  dypit  de  circonstances 
parfois  difficiles.  Cet  aspect  pynalise 
cependant  un  sytyme  de  reconnaissance  avec 
apprent i ssage  pryalable,  fonctionnant  a  par- 
tir  d'une  mythode  globale. 

Le  ryfyrentiel  des  accyiyrations  permet 
de  traduire  les  consequences  physiologiques 
des  efforts  en  fonction  de  leurs  directions 
par  rapport  aux  axes  du  corps.  La  plupart  des 
manoeuvres  ayronautiques  s ' accompagnent 
d1  accyiyrations  +  Gz,  principalement  en 
mission  de  combat,  les  accyiyrations  Gx  sont 
modyrees  ( except e  le  dycollage  catapulte), 
quant  aux  accelerations  Gy,  el  les  sont 
quasiment  inexistantes  [1]. 


-Gz  Figure  1  Referent  iel 
des  accyiyrations. 

On  peut  penser  qu'en  phase  inspiratoire 
l 'acceiyration  Gz  va  s'opposer  aux  mouvements 
d'yiyvation  en  avant  des  cotes  :  le  volune 
inspiry  risque  d'etre  diminuy. 
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lors  de  I'expiration,  l 'acceleration  va 
faire  descend  re  plus  rapi  dement  la  cage 
thoracique,  le  diaphragme  aura  tendance  £ 
s'abaisser  par  rapport  £  sa  position  nominate. 

II  existe  £  notre  connaissance  que  peu 
de  travaux  traitant  du  probl£me  de  l'6locution 
sous  facteur  de  charge,  Les  difficulty  expe- 
rimentales  poshes  par  la  contrainte 
acc£l£rom£trique  et  la  combinaison  du  facteur 
de  charge  4  d'autres  param£tres  (bruit  £lev£, 
masque  £  oxyg£ne,  stress..)  en  sont  sans  doute 
la  cause.  Les  travaux  significatifs  sont  ceux 
de  I'Armstrong  Aerospace  Medical  Research 
Laboratory  (AAMRL)  C33 . 


II  Moyens  d'analyse 


Nous  avons  recens£  4  grandes  families 
d'outils  : 

-  les  m£thodes  frequent iet les  (brute, 
apodis£e  par  Hamming,  calcul  des  moments 
spectraux,  sonogrammes  "bande  large"  et 
"bande  etroite"). 

les  analyses  multi -echel les 
(ondelettes  orthogonales  £  support  compact, 
ondelettes  continues). 

les  decompositions  ceps t rales 
(estimation  du  pitch,  detection  des 
voisements). 

I'analyse  statistique  multi- 
dimensionnelle  (matrice  des  correlations  sur 
une  echel  le  de  Bark,  projections  sur  la  base 
orthogonale  des  vecteurs  propres). 


II. 1)  Decomposition  frequentiel le 


Cette  base  orthogonale  d‘ exponent i el les 
se  r£v£le  excel lente  en  discrimination 
frequentiel le  mais  tr£s  mal  localisee  en 
espace  (ou  en  temps). 

L'apodisation  de  la  tranche 
d'observation  est  souvent  n£cessa  re  pour  une 
bonne  rejection  de  I'effet  parasite  des  lobes 
secondaires.  La  resolution  frequentiel le  A  -3 
dB  est  de  I'ordre  de  l1  inverse  de  la  tranche 
d'analyse  et  peut-etre  relax£e  par 
('utilisation  de  I'adaptativite  du  lobe  de 
directivite  (Algorithme  de  Capon). 

Correiativement,  les  fonctions  orthogo¬ 
nales  de  Haar  (fonctions  binaires)  possedent 
une  bonne  localisation  temporelle  au  detri¬ 
ment  d'une  discrimination  frequentielle 
mediocre. 

La  probiematique  d'analyse  peut  se 
r£suner  ainsi  :  4  partir  d'un  vecteur  r£duit 
de  combinaisons  lin£aires  d'entrees,  il 
s'agit  de  trouver  un  produit  scalaire 
(definissant  ou  non  une  base  orthogonale)  et 
sa  norme  associee,  qui  discriminent  au  mieux 
a  la  fois  les  explosions  temporelles,  les 
structures  harmoniques  et  leur  deplacement 
sous  facteur  de  charge. 

L'analyse  temps-echel le  (propos6e  par 
J.  Morlet  [4])  permet  de  suivre  le  signal  £ 
differentes  §chelles  &  I'aide  d'une  fonction 
analysante  mere  +a  b  (appeiee  aussi 
“ondelette");  cette  fonction  est  param6tree 
par  un  coefficient  de  dilatation  a  et  un 
parametre  de  translation  b.  Cette 
decomposition  dont  le  formalisme  generalise 
les  transformees  d£j£  citees  permet  la  decom¬ 
position  du  signal  sur  des  bancs  de  filtres 
specialement  crees  pour  une  analyse 
d£termin£e. 


L'analyse  de  la  parole  fut  depuis  ces 
premices  regie  par  la  discrimination  spec- 
trale.  Le  formalisme  etabli  £  la  fin  du  si£cle 
dernier  a  suivi  une  evolution,  surtout  dans  sa 
transcription  physique,  analogique  au  debut, 
numerique  ces  dernieres  am£es. 

II  n'en  demeure  pas  moins  que  le  pro- 
bieme  de  I'analyse  du  signal  vocal  de  part  la 
nature  non  statiormaire  du  phenomene  se  r£v£le 
dedicate.  L'uti l isateur  souhaite  en  effet 
avoir  &  la  fois,  une  bonne  discrimination 
temporelle  (plosives,  fricatives)  et  une 
grande  resolution  frequentiel lie  (analyse 
formantique  des  noyaux  vocal iques). 


Depuis  pr4s  de  deux  siecles  le  forma¬ 
lisme  des  series  de  Fourier  •>  permis  de 
decomposer  tout  signal  periodiq'ie  en  une 
somme  de  frequences  pures  (coefficients  de 
Fourier,  spectre  discret).  Pour  les 
phenomenes  quelconques  l' integrate  de  Fourier 
etend  cette  representation  temps- frequence. 

Cette  transformee  est  £gale  au  produit 
scalaire  pour  toutes  les  valeurs  du  temps,  du 
signal  s(t)  par  la  fonction  exp(-j2Tft),  il 
vient  pour  le  spectre  continu  d'amplitude  : 


S(f) 


11a 

r— 


ir,a  s( t).*-**"'  dt 

rJ-T/a 


Cette  probiematique  s'apparente  4  la 
relation  d' incertitude  d'Heisenberg  de  la  m£- 
canique  quant ique  :  le  produit  de  la  mesure  de 
la  position  et  de  la  quantite  de  mouvement 
d'une  particule  est  borr>£e.  Ce  dilemme  resi¬ 
dent  dans  la  necessaire  augmentation  de  la 
bande  (precision  en  distance)  incompatible 
avec  un  allongement  de  la  tranche  temporelle 
(precision  en  Doppler). 

Depuis  la  fin  des  =>nnees  soixante,  des 
algorithmes  de  calcul  rapide  (Cooley-Tuckey) 
ont  permis  de  genera l iser  la  Transformee  de 
Fourier. 


Cette  representation  physique  ne  peut  pas 
etre  £valu£e  systematiquement  de  part  la 
presence  de  1‘ integrate  qui  s'av£re  parfois 
diffici lement  calculable.  On  a  recours  alors 
£  une  approximation  discrete  qui  contient  la 
meme  information  que  le  spectre  continu,  sous 
reserve  que  le  signal  soit  £  spectre  borne 
(interpolation  de  reconstruction  exacte).  La 
transformee  de  Fourier  discrete  d'un  signal 
x(k)  s'ecrit  alors  : 
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La  fonction  X(f)  est  periodique,  de 
p6riode  fe  et  est  g6n6ralement  une  fonction 
complexe  de  la  variable  r6elle  f.  Bien 
qu'uti  Usable  en  l'6tat,  la  vulgarisation  de 
cette  analyse  est  due  6  des  algorithmes 
rap ides  (FFT  :  Fast  Fourier  Transform)  qui 

rgduisent  le  cout  CPU  de  N^  operations  6  N/2 
log 2  N.  L'expression  de  la  transforms  rapide 

est  donn6e  par  : 

*{k)  -  4 

"  B<0 

avec  k  =  0,  N-1 


Tableau  1  :  Elements  de  comparaisons  des 
ponderations. 


La  parole  Sant  un  signal  scalaire  r6el 
quelconque,  sans  n6cessite  de  reduction  de 
debit  par  demodulation  comptexe,  il  en 
r6sulte  une  sym6trie  hermitienne  sur  le 
spectre  d'amplitude  (partie  reelle  paire, 
partie  imaginaire  impaire). 

La  prise  en  compte  du  seul  module 
conduit  6  perdre  une  partie  non  negligeable 
de  l ' information,  il  est  difficile  en  effet 
de  g6rer  convenablement  le  modulo  2t  de  la 
phase. 

Pour  etablir  des  elements  de 
comparaison  entre  les  diverses  analyses 
proposSs  nous  avons  pris  comme  structure 
temporelle  une  tranche  de  25,6  mS  6  ^valuer 
toutes  les  12,8mS  (fe=10  KHz,  fc=4,8  KHz). 

Nous  donnons  figures  2  et  3  un  exemple 
de  spectre  court  terme  (Elocution  “Radar"  6 
1 ,4G)  obtenu  par  une  analyse  de  Fourier 
pond4r6e,  pr6accentu6e  ou  non  par  filtrage 
nunSique  non  recurs  if  du  premier  ordre. 


I.4«"  B««.  rn  foi.4  a.ae  fra.c  elaaalw 


La  figure  3  visualise  la  meme  elocution 
que  pr6c6demment  sans  effet  du  filtrage  en 
tete,  I'energie  est  alors  typiquement 
concentree  dans  les  basses  frequences. 


"MADAR  l.^a"  W«« .  FFT  pond.  ••*««»  ii 


Temps  Frequence 


Figure  2  Analyse  Temps- Frequence  court  terme 
30. 

Nous  resunons  tableau  1  le  n6cessaire 
compromis  entre  eiargissement  du  lobe 
principal  et  niveau  des  sec onda ires 
acceptable. 

La  necessity  du  filtrage  amont 
pr6accentuateur  est,  au  vue  des  traces, 
manifeste  pour  r66quilibrer  I'energie 
spectrale  qui  en  situation  calme  A  tendance  6 
se  local iser  autour  de  800  Hz.  Les  noyaux 
vocal iques  sont  stables  et  clairement  mis  en 
Evidence  par  ce  type  d'analyse. 


Figure.  3  :  Spectre  temps- frequence  court 
terme  3D. 

1 1. 2)  Analyse  formantique 

Pour  mesurer  la  repartition  6nerg6tique 
(essentiellement  autour  des  formants),  une 
approche  dans  l 'estimation  de  cette  frequence 
moyenne,  consiste  a  former  le  periodogramme 
moyenne,  soit  : 

Sxxcf ,  >  =  X(f,-).X*(f,-> 

avec  f,-  =  ik/M 


ou  X(f)  represente  le  spectre  d'amplitude  et 
X*(f )  son  conjuge. 

En  assimilant  la  density  spectrale  de 
puissance  b  la  realisation  d'un  processus 
stationnaire,  il  est  possible  de  calculer 
alors  les  moments  normalises  et  en 
particulier  la  frequence  quadratique  : 


Les  positions  peuvent  etre  alors  evaluees 
dans  des  bandes  pre-6tablies  et  multiples  du 
pas  des  canaux  de  FFT  (Af  *  39,06  Hz);  la 
sommation  est  faite  sur  Ik1,k2]  et  non  pas 
CO ,  N - 1 ]  . 
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la  position  de  la  bande  de  bruit  biaise 
d'autant  l 'estimation  des  formants  ;  pour 
iliminer  ce  biais,  it  suffit  de  remplacer 
Sj^ff)  par: 


Ss(f)  =  Sxx(f) 

lorsque  Sxx(f )max./Sxx(f)  <  K 
=  0,  ailleurs. 


Figure  4  :  Diplacement  formantique  sous 
acciliration  ("Display  Waypoint",  5G) 

11.3)  Sonogramne  liniaire 

Les  outils  pricidemment  explicitis 
relivent  tous  d'une  prise  d* information 
temporelle  par  fenetre  glissante  avec 
recouvrement  de  moitii.  Cette  longueur  doit 
etre  de  dimension  suffisante  pour  une 
estimation  et  un  ichanti 1 lonnage  satisfaisant 
«k J  fondamental.  La  qualiti  de  la  pondiration 
est  beaucoup  mo ins  importante  que  la  largeur 
de  cette  fenitre  d> analyse,  surtout  pour  une 
bonne  estimation  de  la  position  friquentielle 
des  formants.  Une  facon  iligante  de 
visual iser  ces  structures  harmoniques 
consiste  A  tracer  un  sonogramme  dicrit  par  le 
synoptique  figure  5. 


tmmmm 

mm— 


Figure  5  :  Dicomposition  du  Sonogramme 

le  fenetrage  triangulaire  conserve  la 
structure  liniaire  a  la  dicomposition,  la 
largeur  des  bases  fixe  le  type  d' ana  lyse 
(large  bande  lb=  450  Hi,  bande  itroite  lb=  45 
Hz),  les  figures  6  et  7  fournissent  un 
exemple  de  sonogramme  pour  I'ilocution 
“Display"  dit  par  un  bourne  ou  une  locutrice. 


Figure  6  Sonogramme  Bande  Large 


Ce  type  de  visualisation  met  en 
ividence  les  structures  voisies  du  "i"  et  du 
"ay",  'a  sifflante  "s"  est  bien  localisie  en 
haute  friquence  (pattern  caractiristique), 
alors  que  ia  plosive  "p"  tris  courte  sur  le 
traci  temporel  occupe  I'ensemble  des  canaux 
f riquentiels. 

Les  disparitis  entre  ilocution  fiminine 
et  masculine  sont  clairement  mises  en 
ividence  :  diplacement  vers  les  hautes 
friquences  des  noyaux  vocaliques  fiminins, 
doublement  du  fondamental,  6 none i  moins 
inergitique. 

Ill  Analyse  multi-ichelle 
111.1)  Ondelettes  orthogonales 

Pour  compliter  cette  panop l ie  nous 
avons,  A  partir  du  formal isme  de  I'analyse 
multi-ichelle,  ilabori  un  outil  applicatif 
pour  le  signal  de  parole.  Rappel ons 
briivement  les  fondements  de  I'jnalyse.  A 
partir  de  fonctions  ilimentaires  tris 
particulieres,  f one t ion  du  temps  par  un 
indice  b  et  de  la  friquence  par  1/a,  il  est 
possible  de  dicomposer  tout  signal  x(t)  par 
le  produit  scataire  : 

avec 

Ta.b  -  J-.V  C(tHb)/a] 

\/a 

Si  la  base  est  orthogonale,  il  est 
possible  de  reconstruire  le  signal  x(t)  A 
partir  des  coefficients  d'ondelettes  Ca  b 
par  : 

lit)  “JJ^a.b  •  a,b(t)  .da.db 

Nous  avons  choisi  dans  un  premier  temps 
les  ondelettes  de  Daubechies  [5]  a  support 
compact.  Dans  ce  cas  le  facteur  de 
compression-dilation  est  une  puissance  de  2 

(a  =  2m,  m  c  [0,M]),  de  meme  le  facteur  de 
translation  peut  s'icrire  b  =  n.21". 

Les  systimes  de  codage  dans  le  domaine 
friquentiel  sont  i  rapprocher  de  cette 
analyse  qui  consiste  i  fractionner  le  spectre 
en  un  certain  nombre  de  sous-bandes  de 
largeur  Bk  (igales  ou  non),  chaque  sous  bande 
indicie  k  est  riichanti l lormie  &  la  friquence 
de  Shannon,  soit  2  8k.  Les  signaux  issus  de 
chaque  filtre  peuvent  etre  quant ifiis 
diffiremment  en  fonction  de  la  friquence  : 
quantification  accrue  pour  le  fondamental  et 
les  formants,  quantification  plus  grossiire 
IA  oil  I'inergie  est  faible. 
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Le  principe  du  codage  en  sous-bandes 
consiste  &  filtrer  te  signal  par  un  banc  de 
filtre  miroir  quadrature  puis  6  sous 
echantillonner  les  sorties.  La  reconstruction 
se  fait  par  I'addition  de  chaque  sous-bande 
interpose  grace  6  un  filtre  identique  a 
celui  de  la  bande  d'analyse. 


Cette  technique  a  d'abord  et6  implantee 
6  partir  de  filtres  disjoints  contigus  6 
reponse  impulsionnel le  finie.  Elle  a  ete 
ensuite  et endue  par  I'emploi  des  filtres 
miroirs  en  quadrature,  permettant  une 
reconstruction  quasi  parfaite  du  signal.  Le 
filtre  demi-bande,  tel  qu'il  est  d6crit 
figure  7  est  classiquement  un  filtre  lin6aire 
dont  la  fonction  de  transfert  vaut  1/2  b  fe/4 
et  est  antisym6trique  par  rapport  a  ce  point. 

Les  coefficients  hn  sont  nuls  pour  n 
pair  sauf  hg.  Le  gabarit  est  defini  par 
I'ondulation  en  bandes  passante  et  coup6e  et 
par  if  qui  repr6sente  la  largeur  de  la  bande 
de  transition.  Le  nombre  de  coefficients  du 
filtre  requit  en  fonction  du  gabarit  d6sir6 
est  donn6  par  : 

2  1  tm 

M  #  -  lo*  ( - )  .  — 

3  10. 3*  if 

ou  5  =  6i  =  $2  repr6sente  I'ondulation. 

Le  signal  de  parole  etant  correctement 
6chantillonn6,  la  distance  unite  sera  prise 
6gale  au  pas  d' echant  i  Homage.  Si  Sg  est  le 

signal  d'entr6e,les  conditions  sur  l 'analyse 
par  ondelettes  orthogonales  discretes  163 
montrent  que  tous  les  Sj  (pour  j<0)  peuvent 

etre  obtenus  a  partir  de  Sg.  Pour  tout  signal 
f(x),  l 'ensemble  des  echantillons 

sj  -  [  U‘  ■  <  f-  ♦'„>  l„tI 

est  appeie  approximation  discrete  de  f(x)  a 

la  resolution  2J.  Cette  approximation 
discrete  caract6rise  compldtement 

l 'approximation  continue: 

<  r.  >  •  f  f<*>  •  <*>  <>» 

®  J  -4B 

.  f  f(x)  .  (X  -  2*J  .  n)  dx 
•  -• 


Sj  est  alors  la  resultante  de  la 
convolution  de  f(x)  par  echant i 1 1 ortnee 

a  2J.  En  fait,  Sj  correspond  a  un  filtrage 
passe-bas  de  f.  Tous  ies  details  plus  petits 

que  2J  sont  elimin6s. 

En  posant  : 

h(n)  -  <  e"1.  > 

la  reponse  impulsionnel le  d'un  filtre 
discret  H  et  en  notant  H  le  filtre  symetrique 
de  r6ponse  impulsionnel le  : 

h(n)  =  h(-n) 

il  vient  : 

/2j  <  r.  >  -  e  ■  <  f.  ♦i*1 » • >  (2n-kl 

n  k 

ou  <f,g>  designe  le  produit  scalaire  entre 
fonction  f  et  g. 

L'espace  Vj  ainsi  defini  peut  etre  complete 
par  un  espace  Oj  tel  que  : 


Vj  9  Oj  *  Vw 

ou  0J 

est 

l'espace  caracteristique 

des 

detai Is 

presents  a  la  resolution 

mais 

absents 

a  la 

resolution  2J.  Par  le 

meme 

principe 

decrite. 

une 

base  orthogonale  peut 

etre 

La  decomposition  d'un  signal  de  parole 

sur  0J  correspond  a  la  convolution  de  SJ+1 
par  un  filtre  G  en  ne  conservant  qu'un 
echantillon  sur  2;  G  est  defini  par  : 

G(M)  »  o  H  (u  ♦  IT) 

soit  encore. 

Sin)  -  (-I)1'"  .  h(l-n) 

G  est  le  filtre  miroir  de  H  :  c'est  un 
filtre  passe-haut  defini  par  une  reponse 
impulsionnel le  de  type  "ondelette".  H  et  G 
sont  alors  dits  miroirs  en  quadrature  et  la 
compietude  est  assuree. 

La  structure  de  decomposition  du  signal 
echantillonne  6  10  KHz  est  la  meme  que  pour 
I'analyse  temps- frequence.  L'analyse  multi- 
resolution  est  faite  sur  des  trames  de  256 
points  toutes  les  12,8  mS.  La  dilatation  en 

2^  est  effectu6e,  non  pas  en  dilatant  les 
ondelettes  d'analyse,  mais  en  sous 

echant i l lonnant  d'un  facteur  2?  le  signal  a 
arjalyser. 

La  projection  sur  une  base  orthogonale 
n'induit  pas  de  perte  ou  de  redondance 
d' informations.  Pour  une  regularitd  n  de 
fonction  analysante,  le  support  du  filtre 
comporte  2n  valeurs.  Ce  nombre  implique  que 
la  convolution  s'op6re  dans  le  domaine 
temporel  pour  de$  valeurs  inferieures  a  30. 
La  figure  8  dome  la  decomposition 
arborescente  d'une  tranche  de  N  points 
decomposee  sur  2  niveaux;  l 'operation  peut 
etre  renouveiee  autant  de  fois  que  necessaire 
(jusqu'a  log2  N). 


22-7 


Figure  S  :  Decomposition  arborescente 
orthogonale 


A  titre  d'exemple,  nous  montrons  figure 
9  un  exempt e  de  representation  issue  d'une 
decomposition  arborescente  ou  les 
coefficients  d'ondelettes  ont  ete  reorganises 
en  f  one  t  ion  de  leur  rang  de  filtrage 
frequentiel.  Le  niveau  de  decomposition  est 
fixe  A  6  pour  tous  les  canaux  ce  qui  per  met 
de  r6a gencer  I'analyse  temps-echel le  en 
temps- frequence  pour  une  meilleure 
interpretation.  Rappelons  que  par  convention 
nous  par  Ions  de  "coefficients  d'ondelettes" 
te  ou  cheque  valeur  represente  le  resultat  du 
produit  scalaire  assoc > A  aux  parametres  a  et 
b. 


Figure  9  Analyse  Temps-Echelle  Reorganisee 
(Reg.  6,  Dec.  6). 

Nous  montrons  egalement  A  titre 
comparatif  figure  10,  un  trace  similaire  e  la 
figure  9  otj  la  regularite  est  fixee  a  1 
(fonctions  de  Haar).  II  apparait  clairement 
la  mauvaise  rejection  frequentielle  de  la 
batterie  de  filtres.  Augmenter  la  regularite 
de  la  fonction,  consiste  A  tendre  vers  une 
base  "sinusoi'des  amorties"  dont  les 
seconds  ires  se  situent  aux  alentours  de  -30dB 
(Reg.  6).  Sans  effort  d' interpretation 
particulier  nous  pouvons  cependant  noter  que 
les  traits  pertinents  (pitch,  formants, 
sifflante)  sont  bien  mis  en  evidence  par  les 
outils  proposes. 


RAFflR  t>, "  D«co»»r  Oniel 


Ttmps  Frequence 


Figure  10  :  Analyse  Temps-Echelle  RAorganisAe 
(RAg.  1,  Dec.  6} 

III  2)  Ondelettes  continues 

Une  "ondelette"  sera  d'autant  plus 
sensible  aux  transitions  qu'elle  possedera  de 
nombreuses  oscillations.  II  est  possible 
d'imposer  une  condition  d'oscillation  trfes 
forte  qui  est  la  nullite  des  moments  jusqu'A 
l 'ordre  m. 

/  f  (t)  t‘dt«0  0(ki« 

L'admissibilite  constitue  les 

conditions  minimales  pour  qu'une  fonction  ^ 
soit  consideree  comme  une  ondelette  :  ^  doit 
etre  de  moyenne  nulle,  et  le  produit  scalaire 
doit  conserver  la  norme.  Ces  deux  proprietes 
sont  contenues  dans  : 

/  ♦ei2(H) 

\J(|*  (tt»r(«4da><  - 

le  critAre  d'admissibi lite  garantit  la 
nullite  des  moments  A  l' ordre  1.  Nous 
proposons  une  fonction  analysante  de  type 
dArivAe  trois  erne  de  Gaussienne  dont  le 
formalisme  reste  reel  et  qui  grace  A  une 
fonction  de  transfert  asymetrique  permet  une 
bonne  representation  frequentielle;  il  vient 


/"'(*)• 


«*j  o*  r 


le  formalisme  AvoquA  sous-entend  que 
les  fonctions  continues  peuvent  etre 
largement  approximAes  par  un  facteur  de  sur- 
Achanti  llormage  de  8.  A  partir  de 
l 'expression  de  la  transformAe  en  ondelette 
continue,  il  est  possible  de  dormer  A  a  et  b 
une  suite  de  valeurs  rAgul iArement  espacAes. 
Au  terme  des  evaluations  (detection  de  pitch) 
ce  formalisme  est  A  rapprocher  du  filtrage 
adaptA  (convolution  par  la  rAplique).  Sans 
elaboration  supplementaire  pour  l 'instant  sur 
ce  point,  not re  reflexion  fera  bien  sOr 
partie  de  travaux  ultArieurs. 


0  KHz 


Viltur  4*i  CMlhclMll 
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IV  Analyse  Statistique  Multidimensional  le. 

Pour  pouvoir  appliquer  le  formal isme  de 
I'analyse  statistique  multidimensionnelle,  il 
faut  op*rer  une  transformation  des  donnees  de 
mani ire  A  disposer  d'un  tableau 
(paramitres  quantitatifs  .  realisations).  A 
partir  des  canaux  *qui-espac*s  en  sortie  de 
FFT  nous  r*alisons  16  combinaisons  suivant 
l'*chelle  de  Bark  visual is*e  figure  11.  Nous 
avons  represent*  egalement  la  position  des 
divers  rtoyaux  vocal iques  des  3  classes  de  la 
langue  Fran caise. 


Nous  tracons  figure  12  l 'allure  de  la 
repartition  temporelle  des  16  coefficients  de 
Bark  (discr*tis*s  en  amplitude  par  trame  de 
gris).  I1 interpretation  devient  plus  delicate 
du  fait  de  la  non  lin*arite  des  trai tements, 
la  concentration  de  I'energie  reste  bien 
localisee.  A  partir  de  ces  indices  nous 
formons  pour  toute  elocution,  une  matrice  des 
correlations  (analyse  normee).  Apres 
diagonal isation  et  reagencement  des  valeurs 
propres  par  ordre  decroissant  nous  projetons 
le  nuage  A  N  (16)  variables  sur  le  plan 
defini  par  deux  vecteurs  propres 
(generalement  le  2  et  3ieme  ).  La  variance 
est  consid*r6e  cornne  la  dispersion  de 
l' ensemble  des  trames  tempore lies  et  l 'axe  1 
est  celui  pour  lequel  cette  dispersion  est 
maximale,  l 'axe  2  poss6dant  la  meme  propri*t* 
sous  contrainte  d' orthogonal  it*  au  precedent 
et  ainsi  de  suite. 


»0M*  i 
200**  ■ 


Figure  12  :  Trace  femporel  des 
coefficients  de  Bark 


Avant  d* interpreter  les  positions 
relatives  de  deux  variables,  il  est 
necessaire  de  s'assurer  qu'elles  sont  bien 
repr*sentees  et  que  la  proximit*  de  leur 
projection  correspond  6  une  proximit*  reelle. 
Nous  avons  pris  le  cosinus  carr*  de  l 'angle 
reliant  le  plan  6  la  perpendiculaire  passant 
par  la  variable  pour  indicer  cettj  qualite. 

Nous  avons  choisi  une  representation 
simultanee  des  parametres  spectraux  et  des 
trames  temporelles  afin  de  mieux  cerner  les 
elements  de  comparaison  entre  les  variables. 
La  figure  13  repr*sente  un  exemple  de 
projections  simultanees  pour  ('elocution 
"Display"  d*jA  *voqu*e. 


1 1  est  A  noter  que  les  vecteurs  propres 
sont  orthogonaux  mais  ne  sont  que  des 
comb inai sons  tinea  ires  des  variables 
d'entree.  De  ce  fait,  la  projection  des 
trames  sur  le  plan  choisi  doit  etre 
interpret*  uniquement  intra-eiocution.  La 
mise  en  correspondance  de  2  projections 
issues  de  2  diagonalisations  diff*rentes,  n'a 
pas  de  sens  physique  puisque  les  vecteurs 
d*finissant  le  plan  ne  sont  pas  issus  des 
memes  combi nai sons. 

N*anmoins,  il  est  toutefois  possible 
d' interpreter  certains  crit6res  en  se 
confinant  6  une  projection.  Par  exemple,  on 
peut  noter  une  separation  tr*s  nette  sur  le 
plan  (2,3)  entre  les  A  classes  du  fenetrage 
de  Bark  (canaux  BF  de  1  a  4,  et  canaux  HF  14 
6  16). La  trace  acoustique  montre  clairement 
une  separation  en  2  demi  plan  correspondant  * 
la  transition  enf-e  les  deux  syllabes  du  mot 
"Display". 


>r  1  tort  a!  ait*  : 


Figure  13  :  Representation  simultan*e 
sur  les  vecteurs  propres. 

Rappelons  que  12,8m?  s*pare  deux  trames 
cons*cutives.  La  sifflante  "s"  est  locali*s*e 
de  la  trame  T15  *  T19,  A  proximito  des  canaux 
HF.  La  structure  formantique  du  "ay"  est 
visible  A  partir  de  la  trame  T21  jusqu'A  T35 
et  est  beaucoup  plus  proche  des  canaux  BF. 
Globalement  le  facteur  2  repr*sente  l 'axe  BF 
A  droite  et  non-BF  a  gauche  (analogue  a  un 
axe  fr*quentiel  non  lin*aire)  ,  le  vecteur 
propre  indie*  3  repr*sentant  la  HF  dans  sa 
partie  negative. 
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En  fait  nous  avons  constat*  que  seules 
trois  projections  paraissent  g*n*ralement 
int*ressantes  :  il  s'agit  des  3  premiers 
vecteurs  propres. 

Nous  avons  essay*  tout  au  long  de  cette 
definition  des  outils,  de  d*finir  les 
limitations  de  chaque  traitement  propose.  La 
mise  en  evidence  du  compromis  localisation 
temporelle-resolution  frequentiel le  nous  a 
permis  de  recencer  la  sp*cificit*  des 
algorithmes  proposes. 

V  Prise  de  son 

Les  limitations  inherentes  £  la  prise 
de  son  sur  avion  d'armes  sont  dues 
essentiel lemnt  aux  resonances  du  masque 
inhalateur  et  du  circuit  de  gaz  (clapets, 
surpression).  L‘ inspiration,  largement  HF, 
penalise  la  detection  de  fricatives. 

A  ce  stade  de  la  reflexion,  il  faut 
avoir  e  l 'esprit  l 1  imperieuse  n£cessite  de 
conserver  tout  au  long  de  la  phase  de 
reconnaissance,  un  recueil  acoustique  aussi 
proche  que  possible  des  references  acquises 
au  sol.  Toute  disparit*  dans  cette  prise 
d' information  qui  eioignera  les  vecteurs 
d'acquisition  des  vecteurs  cibles,  telle 
qu'une  fuite  du  melange  gazeux,  un  mauvais 
posit iormement  ou  un  deplacement  du  masque 
sous  acceleration,  le  type  ou  la  position  du 
microphone  p£nalisera  d'autant  le  taux  de 
reconnaissance. 


V]  Incidence  du  facteur  de  charge  sur 
la  production  de  parole 

La  methodologie  adoptee  pour  I'etude 
des  retent i ssements  du  facteur  de  charge  a 
consist*  : 


-  dans  un  premier  temps,  *  batir  et 
r*aliser  une  experimentation  preliminaire  en 
centrifugeuse  afin  de  tester  les  param*tres 
dimensionnants  de  I'eiocution  sous 
acceleration.  Les  structures  phonetiques 
simples  :  voyelles,  syllabes,  mots  ont 
constitu*  les  entit£s  .ninimales.  Plusieurs 
mots  (dissyllabes  lex£matiques  ou  non)  et  des 
phrases  completes,  grammatics l ement 
correctes,  faisaient  egalement  partie  du 
panel  et  permettait  de  tester  les  *ventuelles 
modifications  prosodiques.  La  connotation 
operational  le  residait  dans  l 'uti  l  isation 
d'une  dizaine  de  mots  *  caractere 
fonctionnel.  Les  sujets  etaient  au  nombre  de 
6  (5  hommes  et  une  femme). 


dans  un  deuxi*me  temps,  en 
l 'eiaboratoin  d'une  syntaxe  *  A 2  mots  dont 
l 'evaluation,  en  terme  de  taux  de 
reconnaissance,  a  et*  menee  *  1,AG,  3G  et  6G. 
La  longueur  des  phrases  etait  de  8  mots 
maximum,  le  facteur  de  branchement  moyen  de 
l 'arborescence  *tait  de  8.  Cinq  locuteurs 
masculin  et  la  meme  locutrice  que 
pr*cedemment  const i tuaient  le  panel 
d'evatuation. 


VI  1)  Elements  physiologiques 

Nous  avons  procede  apr*s  la  majorit*  des 
tancements  *  un  examen  ORL  des  voies 
aer i ernes  superieures  :  examen  ies  fosses 
nasales  et  des  sites  anatomiques  mobiles. 
L' object  if  de  ces  examens  etait  d'appr*cier 
la  coloration  des  muqueuses,  avec  sous 
jacente  I'hypothese  que  les  efforts  vocaux 
sous  facteur  de  charge  entrainent  une 
congestion  (rubefaction)  des  muqueuses  qui 
tapissent  les  organes  mobiles.  Un  examen  au 
miroir  larynge  a  *te  pratique  avant  et  apres 
lancement. 

Nous  donnons  tableau  2  les  resultats  des 
quotations  colorim*triques  pour  tous  les 
sujets.  Les  trois  niveaux  d" appreciation 
(o  :  pas  de  changement,  +  :  leger  changement, 
++  :  tres  net  changement)  vont  du  rose  au 
tr*s  rouge. 


SUJET 

PIECES  FIXES 

PIECES  MOBILES 
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Tableau  2  :  Modifications  conges tives  des 
muqueuses 

Dans  la  mesure  ou  le  larynx  n'est  li*  a 
aucun  os  et  se  trouve  tenu  par  une  serie  de 
muscles,  il  aura  tendance,  sous  acceleration, 
a  descendre  (notamment  pour  les  voyelles 
poster ieures).  D'autre  part,  le  basculement 
mecanique  du  cartilage  thyroi'de  sur  le 
cartilage  cricoide  aura  pour  effet 
d'accroitre  la  tension  des  cordes  vocales  ce 
qui  produira  une  augmentation  de  la  frequence 
fondamentale. 

Enfin,  en  ce  qui  concerne  les  cordes 
ocales  deux  hypotheses  plus  deiicates  a 
verifier  peuvent  etre  emises  :  d'une  part,  le 
fait  que  l 'acceleration  s'effectue  dans  un 
sens  qui  s'oppose  *  I'ecoulement  de  I'air 
pourrait  modifier  la  forme  glottique;  d'autre 
part,  le  temps  n*cessaire  a  ('abduction  et  £ 
l 'adduction  des  cordes  vocales  qui,  en  raison 
d'une  mobilite  limit*e,  devrait  augmenter. 

Les  cavites  situees  au  dessus  du  larynx 
constituent  des  r*sonateurs  supraglottiques 
(cavit*  pharyngale,  buccale  et  nasale).  Le 
volume  et  la  forme  de  ces  cavit*s  pourraient 
etre  re duit  par  les  tensions  exerc*es  sur  les 
;;ff*rents  muscles,  les  resonances  auront 
alors  tendance  a  se  deplacer  vers  les  hautes 
frequences. 

La  langue  est  largement  affect*e  par 
l 'acceleration.  Les  constrictives  dentales 
demandant  son  posi t ionnement  precis  sont  les 
premieres  touchees. 

L'ouverture  de  la  mandibule  rend 
difficile  l 'articulation  des  consonnes 
bi labiales. 
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VI. 2)  Corr6lats  acoustiques 

La  definition  de  modules  de  variation 
acoustique  est  indispensable  pour  introduire 
des  algorithms  de  compensation  permettant 
d'am6l  iorer  des  scores  de  reconnaissance  en 
ambiance  avionique.  L'analyse  des  changements 
qui  interviennent  doivent  etre  suffisament 
reproduct ibles  pour  pouvoir  degager  des  axes 
d1  invariance. 

La  premiere  phase  (items  langagiers  non 
operational)  a  mis  en  evidence  : 


Hi  ♦  r>, 

MOO^  \ 

FI—  . 

\ 

)  000  - 


une  augmentation  notable  de  la 
frequence  du  fondamental  (pouvant  passer  de 
130Hz  e  200Hz  pour  un  locuteur,  voire  doubler 
pour  la  locutrice  et  depasser  300  Hz).  Nous 
domons  figure  14  un  exemple  caracteristique 
de  ce  phenamene  pour  I'eiocution  "Radar"  a 
1,4G  et  60  (Sonogramme  bande  large). 

-  un  renforcement  notable  des  hautes 
frequences,  rehausse  par  le  pr6accentuateur 
nuntrique, 

un  accroissement  du  niveau 
energetique,  dont  I’effet  est  inhibe  par  les 
caracteristiques  de  la  chaine  de 
paraaietrisation, 

-  une  variabilite  accrue  pour  les 
formants  les  plus  hauts.  Cette 
caracteristique  est  emoussee  par  le  fen6trage 
de  Bark  (Cf  figure  12). 


Figure  14  :  Variabilite  de  l ’elocution 
"Radar"  sous  facteur  de  charge 

La  variabilite  des  structures 
formantiques  s’explique  assez  simplement  par 
un  raisomement  d’acoustique  mecanique.  La 
figure  15  visualise  les  differentes  positions 
des  formants  FI  6  F4  en  f one t ion  d’ une 
constriction  de  lcm/0,5  cm2.  En  relation  avec 
le  deplacement  de  l ’occlusion  par  rapport  6 
I’extremite  gauche  du  tube  (simulant  les 
levres)  les  deplacements  des  formants  peuvent 
apparaitre  similaires  ou  au  contraire 
antinomiques. 


I?  kill'd)  I  k  /  •  I 
o  l  1  5  4  }  l  7  i~em 

Figure  15  Resonance  acoustique  d’une 
occlusion  d’apres  (7) 

En  considerant  l ’acceleration  conine  un 
facteur  qui  vient  se  surimposer  et  qui  a 
tendance  a  r6duire  les  volumes  (deplacement 
de  l ’occlusion  vers  la  droite),  il  est  alors 
possible  d’obtenir  sur  un  cas  d’occlusive 
dentate  (t/d)  une  augmentation  de  F2 
correiativement  a  une  diminution  de  F3;  de 
meme  sur  une  occlusive  compacte  (k/g)  le 
phenomene  inverse.  Ces  ph6nom6nes 
assimi tables  a  des  battements  induisent  une 
versatilite  accrue,  li6e  au  facteur  de 
charge,  pour  F3  et  F4. 

Nous  avons  egalement  constate  que  le 
mouvement  ventilatoire  est  eorr6 16  de  manifcre 
6troite  a  l ’experience  du  sujet  vis  6  vis  des 
accelerations  specifiques  de  la  centrifugeuse 
et  du  dialogue  vocal.  11  peut  etre  generateur 
d’ expirations  n6fastes  (au  beau  milieu  d’une 
phrase)  lorsque  le  locuteur  ne  maitrise 
suff isamment  pas  son  rythme  prosodique.  Un 
apprentissage  prealable  (clear  speech) 
ameiiore  sensiblement  ce  debit  d’eiocution  en 
particulier,  et  le  taux  de  reconnaissance  a 
terme. 

VII  Incidence  du  facteur  de  charge  sur 
un  systdme  de  reconnaissance  de  parole 

VII  1)  Campagne  d’essais  proprement 

dite 

Elle  s’est  deroulee  en  mai  et  ju'n 
1991.  Le  panel  representatif  des  sujets  etait 
constitue  de  5  locuteurs  et  d’une  locutrice 
equipes  d’un  masque  Ulmer  87  (micro  Silec 
S  4045)  avec  casque,  r6gulateur  d’oxygdne  et 
pantalon  anti-G  fonctionnel.  Tous 
appartiennent  au  personnel  du  CEV. 

Le  chronogramme  d’une  sceance 
journalise  est  fournie  figure  16. 


Figure  16  :  Prof i l  8Cc6l6rom6trique 
journal ier. 
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t 1  apprent i ssage  nEcessai re  pour  la 
mEthode  de  reconnaissance  envisagee  se 
dEroulait  systEmatiquement  en  une  fois,  et  en 
prEmice  E  tout  lancement.  La  centrifugeuse 
Etant  E  t 'arret  deux  tistes  sont  enoncees  : 
cetle  des  mots  i soles ,  puis  celle  des  mots 
enchainEs. 

La  liste  servant  de  support  § 
t 'evaluation  du  systEme  de  reconnaissance 
Etait  const i tuE  par  une  sErie  de  100  phrases 
scindEes  en  14  listes  de  30  secondes  .  A 
raison  de  6  paliers  journaliers  A  1,4G,  3  a 
3G  et  3  A  6G,  la  totality  des  listes 
(sequence  random sEe)  requiert  4  jours  par 
individu.  L'analyse  du  signal  s'opErait  de 
maniEre  concotni  tante,  un  suivi  medical 
permettait  de  cerner  les  retent issements 
physiologiques. 

Les  phrases  A  prononcer  s'aff ichaient 
sur  un  Ecran  place  dans  la  centrifugeuse; 
I'EnoncE  n'Etait  done  pas  spontanE. 

Pour  une  exploitation  plus  fine  des 
rEsultats  nous  avons  sc i nde  les  plateaux  £ 
1,4G  ent recoupes  des  plateaux  A  3G  et  ceux 
ent recoups  des  plateaux  E  6G.  Nous  avons 
done  in  fine,  4  conditions  accElEromEtriques 
component  chacune  100  Elocutions. 

Nous  dressons  pour  chaque  cas  un  taux 
de  reconnaissance  avec  : 

-  une  paramEtrisation  cepstrale  sur  une 
Echelle  de  Bark, 

-  une  mEthode  de  comparaison  dynamique 
(DTU  :  Dynamic  Time  Warping), 

-  une  ou  plusieures  rEfErences  par 

mots, 

une  segmentation  manuelle  ou 
automatique,  des  items  £  reconnaitre, 

-  les  rEfErences  propres  au  locuteur. 

A  partir  des  premiers  rEsultats  nous 
avons  pu  constater  que  le  taux  de 
reconnaissance  obtenu  avec  une  segmentation 
manuelle  Etait  pratiquement  independant  du 
facteur  de  charge  pour  la  syntaxe  de  42  mot9 
envisagEe  (Cf.  figure  17). 

I  m  I  '  «■/»» 
■  It  NF/wt 

Taux  da  aacentaUaanea  |P  I  ' 


Figure  17  :  Taux  de  reconnaissance  sur  100 
Elocutions 

(5  locuteurs,  seg.  manuelle). 


VI l  2)  Durcissement  de  l 'algorithmic 

Ce  durcissement  tel  que  nous  l 'avons 
initialement  prEvu  peut  Etre  dEcoupE  en 
quatre  phases  distinctes  : 

une  paramEtrisation  cepstrale  £ 
partir  de  12  coefficients  autorEgressifs  (LPC 
:  linear  predictive  coding)  venant  supplEer 
la  paramEtrisation  classique  sur  les 
structures  voisEes, 

-  une  dEtection  de  parole  robuste  au 
facteur  de  charge  per  sttant  de  relaxer  les 
contraintes  classiques  dues  aux  positions 
temporelles  de  I'alternat,  souvent  trEs 
approx i  natives. 

-  un  filtrage  nunErique  amont  utilisE 
pour  compenser  les  disparitEs  de  la 
rEpartition  EnergEtique  sous  accElEration, 

•  un  assouplissement  de  la  syntaxe 
prennant  en  compte  des  pauses  optiormelles  et 
des  rEfErences  de  respiration. 


VII  2  1)  Codage  prEdictif 

Nous  avons  choisi  I'algorithme  de  Durbin- 
Levinson  par  bloc,  ceci  afin  de  faciliter  la 
convergence  et  la  stabilitE  des  coefficients. 

Si  l 'on  suppose  la  linEaritE  du  sytEme 
de  parole  lors  de  l 'Emission  des  structures 
voisEes,  celles-ci  peuvent  etre  modElisEes 
par  ui  filtrage  linEaire  autorEgressif  (AR). 
Le  nombre  de  poles  du  modEle  correspond 
physiquement  au  nombre  de  rEsonateurs  mis  en 
cascade. 

Nous  avons  voulu  : 

Evaluer  I'apport  d'un  calcul  de 
distance  entre  les  cepstres  bat  is,  soit  sur 
une  analyse  frEquentielle  classique,  soit  sur 
une  estimation  autorEgressive  du  signal  de 
parole, 

-  calculer  des  paramEtres  de  concor¬ 
dance  pour  un  algorithme  de  vraisemblance 
acoustique  de  la  phrase  reconnue, 

-  fournir  au  besoin  les  coefficients 
d'un  filtre  dEbruiteur  par  rEjection 
transverse  adaptative. 

L 'algorithme  de  Durbin- Levinson  rEsoud 
de  maniEre  rEcursive  les  Equations  de  Yule- 
Walker  synthEtisEes  par  : 

R.a  *  P 

ou  R  est  la  matrice  des  corrElations  , 
a  le  vecteur  des  coefficients  autorEgrEssifs 
{ag=1>  et  P  est  le  vecteur  nul,  sauf  pour  la 

premitre  composante  qui  est  Egale  a  la 
puissance  de  bruit.  L'erreur  dEcroit 
continuement  lorsque  I'ordre  augmente. 
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Nous  calculons  tous  les  12,8ms  12 

coefficients  (moddle  a  6  rlsonateurs)  sur  un 
bloc  de  25,6ms.  Nous  donnons  figure  18  un 
exempt e  de  density  de  puissance  AR  d'un  "0" 
pr6accentue. 


(12  coefficients). 

la  majority  des  erreurs  residant  dans 
la  reconnaisance  de  digits,  nous  6tablissons 
des  Elements  comparatifs  entre  cepstres 
classiques  et  cepstres  LPC  pour  les  10 
premiers  chiffres  de  la  langue  Franqai se  en 
laboratoire  : 


Cepstres  Amadeus 
Ftotttnt 

Cepstres  LPC 
Levinson 

fttsultats 
gtobtui 
item  *1"  i  ’9- 

*4/88 

ELS 

63/M 

TUk 

sans  item 
•4\  -8".  -9" 

47/60 

7LS 

49/60 

ULS 

item 

"4*,  -8-.  “9- 

17/28 

41_S 

14/28 

HUS 

Tableau  3  :  R6sultats  de  la  reconnaissance 


Les  resultats  6tant  sensiblement 
comparables  entre  les  deux  techniques,  il 
nous  a  paru  inopportun  de  substituer  la 
param6trisation  cepstrale  par  une 
param6trisation  LPC.  La  necessite 
d'harmoniser  les  coefficients  pour  les 
int6grer  dans  le  calcul  de  la  distance  entre 
references  et  trame  courante  induit  une 
refonte  de  la  chaine  de  parametrisation  que 
nous  pr6voyons  6  moyen  terme.  Cette  refonte 
doit  fusionner  des  techniques  anciennes 
(analyse  temps-f r6quence)  et  des  techniques 
6mergentes  (analyse  temps-6chel le). 

VII  22)  Detection  d'activite 

L'uti lisation  de  la  commande  vocale  en 
avions  d'armes  est  r6git  par  I'utilisation 
d'un  alternat  qui  vient  indiquer  au  systfrme 
la  demande  de  la  fonction  reconnaissance.  Cet 
alternat  temporellement  approx i mat  if  est  chi 
ressort  de  l  'utilisateur  et  done  tr6s 
versatile.  II  est  soumis  6  l 1 apprent i ssage  au 
facteur  de  charge,  aux  ambiances  de  stress 
qui  lui  soot  lies  et  au  degr6  de  motivation 
du  pilote  vis  i  vis  de  la  commande  vocale. 

A  pertir  de  cette  premiere  localisation 
temporelle  et  conformement  aux  r6sultats  de 
la  figure  17,  indiquant  la  Constance  des 


performances  de  reconnaissance  lorsque 
les  items  sont  segments  6  la  main,  nous 
avons  contruit  une  algorithmic  de  recherche 
de  I'enonce. 

Apres  de  multiples  architectures 
envisagees,  notre  conviction  s' est  forgee 
autour  d'une  detection  locale  autour  du  mot  a 
partir  de  l' indication  grossi£re  de 
I'alternat.  Le  synoptique  final  peut  etre 
resume  figure  19  : 


Figure  19  :  Detection  de  parole 


A  partir  de  ce  canevas  general,  chacune 
des  macro-fonctions  peut  etre  etablies  par 
une  multitude  de  methodes  qui  recdlent  toutes 
des  specif icites. 

VII  23)  Detection  de  voisement 

L'ossature  de  notre  methode  est 
articuiee  autour  une  localisation  efficace  et 
robuste  du  noyau  vocal ique.  Nous  avons  pour 
ce  faire  retenu  deux  principes  : 

-une  methode  bas6e  sur  la  fonction  de 
correlation  oil  I'on  calcule  : 

C(t)  ■  *£  4»>  «»**>  ■ 1  ’  -  K 

•-I 


-  une  methode  utilisant  les  differences 
moyennes  de  la  correlation  (AMDF  :  Average 
Magnitude  Difference  Function)  ou  I'on  evalue 
l 'expression  : 

AM-* 

W)  •  £  I  *»)  -  | .  i  -  0.  1. ...  K 

•  •I 

Les  figures  20a  et  20b  caracterisent 
les  resultats  obtenus. 
x  10*  Correlation 


6U  J 


decalaee  en  Nbre  de  Pts 

Figure  20  a)  Correlation  d'un  "0". 
(6  Trames  voisecs) 
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xlO*  AMDF 


Dccalage  en  Nbrc  de  Pb 


Figure  20  b)  AMDF  d'un  "0" 

(6  Trames  voisEes) 

Alors  que  le  maximun  de  la  correlation 
est  bien  local  isE  et  faci lenient  detectable, 
l 'AMDF  (precise  sur  la  position  de  ces 
minimuns)  requiert  une  expertise 
supplEmentaire  pour  traiter  l 'aspect  physique 
(Elimination  des  minimuns  locaux).  D'autre 
part,  une  information  d'Energie  mutuelle  a 
disparu  par  diffErence  :  le  traitement  du 
minimun  absolu  est  en  fait  mo  ins  facilement 
exploitable. 

La  dEtection  de  voi semen t  proposEe  peut 
etre  schEmatisEe  figure  21. 


****  »,  ■ 


Les  scores  obtenus  soot  rEsumEs  dans  le 
tableau  4 


Tableau  4  Incidence  du  filtrage  amont 


Rep.  AmpL  fibre  2c me  orctre  bl ■-!.  b2--0J 


4j- 


Fieq. 


Figure  22  :  Type  de  filtrage 

Nous  en  dEduisons  I'impErieuse  nEcessitE  de 
prEaccentuer  la  phase  de  reconnaissance. 


VIII  Assouplissenent  de  la  syntaxe 


Figure  21  DEtection  de  voisement. 

L' interpolation  pour  estimer 
prEcisement  le  maximun  n'est  pas  nEcessaire  E 
condition  de  garder  un  dEbit  constant  en 
sortie  de  dEcimation. 

le  filtre  logique  Elimine  les  trames 
voisEes  isolEes  et  opEre  une  agrEgation  des 
divers  noyaux  vocal iques.  Cette  expertise  est 
fonction  de  l 'antErioritE  du  message  . 

Le  dispositif  de  dEtection  finale 
intEgre  aprEs  localisation  du  voisement  une 
detection  de  fricative  par  filtrage 
frEquentiel  et  seuillage  EnergEtique. 


Afin  de  compenser  la  variabilitE 
d'Elocution  sous  accElEration,  liEe  au 
facteur  humain,  nous  avons  bati  deux  syntaxes 
rEguliEres  (SI  et  S2).  SI  reprEsente  la 
syntaxe  brute;  S2  rEfErence  la  syntaxe 
modifiEe  par  I'ajout  de  trois  types  de 
reprEsentants  : 

-  des  rEfErences  de  pause, 

-  des  rEfErences  de  respiration, 

-  une  coarticulation  sans  pause  (par 
exenple  entre  le  mot  de  comma  ride  et  les 
digits). 


Nous  donnons  les  rEsultats  globaux  pour 
VII  4)  Filtrage  amont  .  le  locuteur  5 
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Le  seul  handicap  de  cette  astuce,  qui 
relaxe  les  limitations  li£es  a  la  detection 
de  parole  automatique  est  d'augmenter 
sensiblement  le  nombre  d'arcs  syntaxiques  et 
par  deli  le  temps  CPU  de  la  phase  de 
reconnaissance  des  formes  (Cf.  tableau  6). 


Facteur  de  branchemem 

Longueur  de  phrases 

— 

Emropte 

min 

moy 

max 

min 

moy 

max 

SI 

1 

7,7 

22 

3 

7.9 

8 

1.89 

Syntaxe 

S2 

1 

11.8 

24 

: 

7.2 

8 

2.71 

Tableau  6  :  Elements  de  comparaison  entre  SI 
et  S2. 

A  ce  stade  de  la  reflexion,  il  nous 
parait  prejudiciable  de  binariser  une  telle 
syntaxe  au  risque  d'augmenter  fortement  le 
nombre  de  solutions  potentiel les.  Le  choix  de 
plusieurs  syntaxes  regul i£res  emboit£es  nous 
parait  etre  plus  appropri£  pour  traiter, 
suivant  les  situations  acc£l£rom£triques,  les 
limitations  physiologiques  et  les  besoins 
op£rationnels  diff£rents. 


VIII  Conclusion 

Nous  avons,  a  parti  r  d'une 
experimentation  pr£liminaire,  etabli  les 
effets  physiologiques  et  les  corr£lats 
acoustiques  de  l 'elocution  sous  facteur  de 
charge.  A  partir  des  tentatives 
d* interpretations,  nous  avons  souligne 
l 'absence  d' invariance  des  deformations  et 
les  rententissements  pour  un  durcissement  de 
l 'algorithmic. 

Sur  des  enregistrements  "centrifugeuse" 
(facteur  de  charge  seul),  il  est  facile 
d'obtenir  des  scores  comparables  a  une 
reconnaissance  manuelle.  Il  suffit  pour  ce 
faire  de  disposer  d'une  detection  de 
voisement  et  d'une  expertise  fiable,  puis  de 
prendre  une  heuristique  d'une  dizaine  de 
trames  avant  et  apres  le  noyau  vocal ique. 
L' absence  de  bruit  conduit  £  un  taux  de 
reconnaissance  qui  n'est  pas  penalise  par  ce 
procede.  En  configuration  operational  le,  il 
faut  etendre  cette  demarche  par  une  reflexion 
globale,  £  savoir  l 'aspect  d£bruitage  et 
confirmation  des  trames  de  bruit.  Le 
debruitage  [8]  ne  doit  pas  etre  vu  uniquement 
comme  un  facteur  ameiiorant  la 
reconnaissance,  mais  surtout  comme  un  outil 
d'aide  £  une  detection  robuste,  done  par  voie 
de  consequence  £  un  taux  de  reconnaissance 
accru. 

La  future  £tape  est  la  validation  de 
l 'algorithmic  sur  une  syntaxe  £  155  mots 
(type  Rafale  0)  en  centrifugeuse.  Au  terme  de 
cette  phase,  il  sera  possible  de  transposer 
l 'algorithmic  pour  un  portage  sur  avion  banc 
d'essai,  afin  d'en  £valuer  les  performances 
op£rationnelles  en  situation  r£elle. 
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1.  SUMMARY 

Current  thinking  on  the  aircraft  man-machine 
interface  focuses  primarily  on  the  cockpit.  My 
proposed  approach  will  view  the  mission 
planning/mission  rehearsal  and  the  aircraft  as  an 
integral  system  that  can  perform  three  functions:  (1) 
dynamically  adjust  itself  to  the  particular  pilot 
response  in  the  planning/rehearsal  stage,  (2)  adjust 
itself  during  the  strike  based  upon  the  scenario 
encountered,  and  (3)  adjust  itself  after  each  mission 
to  enhance  planning  for  follow-on  missions.  The 
objectives  are  to  improve  strike  effectiveness  and  to 
shift  portions  of  the  pilot's  work  load  from  the  attack 
to  the  planning  phase  of  the  strike. 

This  concept  can  be  implemented  using  existing  and 
emergent  technologies.  Aircraft  now  coming  on  line 
are  equipped  with  removable  disks  that  are  used  to 
load  misaion  specific  data  for  the  avionics  and 
weapon  systems  (maps,  ELI  NT  flies,  route  plans, 
navigation  data,  target  coordinates,  etc.).  Mission 
planning  systems  are  being  configured  not  only  to 
provide  these  data,  but  also  to  evaluate  post  flight 
data  from  flight  recorders.  Mission  planning  and 
rehearsal  with  the  resulting  aircraft  mission  data 
package  will  provide  more  inputs  for  the  aircraft 
mission  computer  that  will  eventually  fly  the 
mission.  Inter-aircraft  computer  communications 
will  support  adaptively  optimizing  the  strike  based 
upon  the  threat  and  target  conditions  encountered  and 
the  success  of  the  strike  to  that  point  in  time 
(adaptive  mission  control).  Pilot  intervention  may 
only  be  required  for  aircraft-to-aircraft  combat, 
freeing  the  pilot  to  attend  to  weapon  aimpoint 
selection  tasks  (where  the  automatic  systems  have 
low  confidence  in  their  selections)  or  for  unplanned 
contingencies.  This  paper  describes  the  mission 
planning/aircraft  system,  the  notional  aircraft 
avionics  for  adaptive  mission  control,  and  the 
implications  for  the  man-machine  interface. 

2.  INTRODUCTION 

The  future  is  certain— things  will  change.  The 
systems  developed  today  for  service  10  years  from 


now  will  enter  service  in  a  different  world.  The  trend 
is  toward  a  military  half  the  current  size— fewer 
aircraft,  fewer  carrier  decks,  fewer  forward  bases— to 
face  enemies  with  weapons,  including  ballistic 
missiles,  fighters,  and  air  defenses,  much  more 
capable  and  mobile  than  currently  exist.  And  yet  the 
desires  and  expectations  will  be  for  a  vastly 
improved  capability  to  project  force.  The  desires 
derived  from  Desert  Storm  are  a  good  example— one 
target/one  warhead;  no  fratricide;  real  time  utilization 
of  intelligence  data;  higher  sortie  rates  for  manned 
aircraft;  joint  force  coordination;  effective  ballistic 
missile  launcher  targeting;  no  loss  of  people; 
minimum  collateral  damage;  etc.  How  will  we  provide 
these  dramatic  improvements  with  only  half  the 
resources?  A  vision  is  forming  that  the  dramatic 
improvement  will  come  about  through  the  tactical 
utilization  of  all  our  reconnaissance  data  and  real 
time  adaptive  coordination  of  weapon  platforms. 
Cooperative  targeting,  netted  fire  control,  terrain 
reference  navigation,  over  the  horizon  targeting,  and 
sensor-to-shooter  data  links  are  becoming  the  buzz 
words.  To  ensure  that  plot  overload,  as  illustrated  in 
Fig  1,  is  not  the  weak  link  in  the  chain,  mission 
planning  and  the  aircraft  must  be  developed  as  an 
integral  system. 

In  the  next  section,  an  overview  of  the  advanced 
aircraft  man-machine  system  is  presented.  The 
following  sections  present  notional  components  of 
mission  planning  and  aircraft  subsystems.  Lastly,  I 
will  summarize  our  development  approach. 

3.  ADVANCED  AIRCRAFT  MAN- 
MACHINE  SYSTEM 

There  are  three  goals,  as  illustrated  in  Fig  2,  for  this 
notional  aircraft  man-machine  system:  (1)  improved 
discriminate  use  of  force  for  reduced  collateral 
damage,  improved  target  kill,  and  elimination  of 
fratricide;  (2)  improved  effectiveness  in  terms  of 
reduced  timelines  to  achieve  missions,  increased 
platform  utilization  in  non-traditional  roles, 
reduction  in  the  number  of  aircraft  and  weapons 
necessary  to  accomplish  a  mission,  and  improved 
survivability;  and  (3)  reduced  pilot  work  load  to 
enhance  mission  adaptability,  shorten  mission 
response  times,  and  increase  the  number  of  sorties  a 
pilot  is  able  to  fly. 

As  witnessed  in  the  Gulf  War,  daily  strike  plans  were 
based  on  an  Air  Force  mini-SIOP  (Single  Integrated 
Operational  Plan)  type  of  listing  (Ref  1).  From  this 
integrated  plan,  generated  for  the  regional 
commander  in  chief  (C3NC),  the  multi-national  forces 
and  various  services  developed  the  detailed  mission 
plans.  As  pressure  for  response  time  and 


effectiveness  increase,  and  the  resources  available  to 
a  theater  commander  decrease,  the  function  of 
mission  planning  will  assume  a  real  time  adaptive 
nature.  And  direct  control  of  aircraft  for  time  critical 
targeta/situations  will  be  assumed  by  the  theater 
commander,  as  illustrated  in  Fig  3.  Some  of  the 
traditional  mission  planning  functions,  such  as  route 
and  attack  planning,  will  have  to  shift  from  airfields 
and  carriers  to  the  aircraft.  Other  functions  such  as 
queuing  for  cooperative  targeting  of  short  dwell 
targets,  resource  reallocation  (tankers,  combat  air 
patrol,  defense  suppression,  etc.),  and  deconfliction 
will  be  handled  in  real  time  by  the  CINC. 

An  architecture  for  the  mission  planning/aircraft 
system  that  could  facilitate  real  time  dynamic 
reallocation  is  illustrated  in  Fig  4.  The  key  feature  of 
the  architecture  is  that  mission  planning  and  the 
aircraft  are  considered  a  system.  And  as  a  system,  the 
components  of  each  can  be  synergistically  tailored 
to  reduce  the  pilot  work  load  while  providing  more 
capability. 

The  single  most  important  feature  of  the  system  is 
the  rapid  mission  assessment  function:  the  ability  to 
verify  target  kills  on  board  the  aircraft  and  to  make 
use  of  that  data.  In  older  to  husband  our  weapon  and 
pilot  resources  we  only  want  to  kill  a  target  once.  "It 
was  only  when  the  Iraqis  started  to  paint  bomb  holes 
onto  undamaged  HASs  [hardened  aircraft  shelters] 
that  the  Coalition  realized  that  it  had  no  record  of 
which  shelters  had  been  attacked  and  which  were 
really  undamaged"  (Ref  2).  For  many  targets,  an 
unverified  kill  requires  another  mission. 

Other  features  include  the  following. 

*  Rapid  strike  planning  optimizes  the  strike, 
including  individual  routes,  terminal  area  acquisition, 
and  weapon  release  plans,  given  the  targets  and 
resources  available. 

*  Contingency  planning  considers  operations  being 
conducted  adjacent  to  the  mission  operation  area. 

*  Mission  control  and  rehearsal  package  preparation 
includes  automated  flight  options  for  rehearsal  either 
in  ground  simulators  or  in  the  cockpit. 

*  Navigation  and  precision  registration  for 
registration  with  other  allied  forces,  registration 
with  absolute  earth  coordinates,  and  registration  with 
the  target. 

*  Cockpit  mission  rehearsal,  attack  cueing,  and 
assessment  through  multimedia  display  of  mission 
rehearsal  data,  real  time  targeting  sensor/data  base 
registration,  or  a  quick  look  assessment  of  an  earlier 
engagement. 

*  Adaptive  mission  control  for  enroute 
reoptimization  of  strike  resources,  routes,  and 
weapon  release  based  upon  current  conditions. 


*  Data  representation  and  storage,  including  data 
compression  and  decompression  for  cockpit 
utilization,  and  capture  of  significant  mission  data 
for  later  assessment. 

4.  MISSION  PLANNING  SUBSYSTEM 
Mission  planning  must  be  highly  automated  and 
efficiently  and  effectively  interfaced  to  the  aircraft 
and  crew.  It  must  be  developed  with  the  pilot 
foremost  in  mind  and  with  the  asaumption  that  all 
functionality  will  eventually  end  up  in  the  aircraft. 
The  recurring  emphasis  on  "rapid"  throughout  this 
section  of  the  paper  is  not  accidental.  "Timely 
results"  must  be  produced  for  the  planning  systems  to 
be  used  widely  in  combat  situations.  The  four 
functions  defined  for  this  notional  planning  system, 
as  mentioned  above,  are:  (1)  rapid  strike  planning, 
(2)  contingency  planning,  (3)  rapid  mission 
assessment,  and  (4)  mission  control  and  rehearsal 
package  preparation  and  mission  rehearsal. 

Rapid  strike  planning  will  start  with  a  data  base 
comprised  of  a  growing  number  of  data  items, 
including  the  operational  plans,  maps,  charts,  GPS 
almanac,  SIGINT,  IMINT,  etc.,  orders  of  battle, 
weapons  characteristics,  aircraft  characteristics, 
threat  lethalities,  sensor  characteristics,  signatures, 
target  vulnerabilities,  etc.  Some  of  these  could  be 
updated  through  real  time  inputs.  The  first  step  in 
doing  anything  "rapid”  with  this  growing  mountain 
of  data  will  be  automated  data  fusion  and 
understanding,  as  illustrated  in  Fig  S.  What  things 
are,  where  they  are,  what  they  look  like  in  selected 
parts  of  the  spectrum,  their  vulnerabilities,  lethality, 
and  value,  and  anything  else  we  want  to  know  about 
them,  must  be  automatically  correlated  or  extracted 
and  put  in  a  usable  form.  The  RADIUS  project  at 
Defense  Advanced  Research  Project  Agency  (Ref  3) 
for  automated  object  recognition,  three-dimensional 
feature  extraction,  and  change  detection  from  two- 
dimensional  and  stereo  pair  imagery,  is  an  excellent 
example  of  growing  technical  capability  in  this  area. 

The  second  step  in  rapid  strike  planning  is  to  build  a 
coordinated  attack  plan,  allocating  appropriate 
aircraft  and  weapons  to  appropriate  targets,  and 
providing  coordination  for  defense  suppression, 
jamming,  tanking,  etc.  Products  currently  exist  that 
do  resource  allocation  and  optimization,  such  as 
SCTs  FLAP  system  (Ref  4),  but  optimization  of 
complex  resource  allocation  is  computer  intensive, 
especially  if  we  need  sensitivity  analysis  to  show  the 
operator  what  the  trade-offs  are.  Work  at  the  Jet 
Propulsion  Laboratory  is  addressing  this  issue 
through  the  fabrication  of  specialized  neural  network 
chips  specifically  tailored  to  rapidly  solve  this  class 
of  problems  (Ref  5). 

The  third  step  is  to  determine  optimum  aircraft  routes 
for  ingress  and  egress.  Again,  systems  have  existed 
for  years  to  do  auto-routing  of  cruise  missiles,  taking 
into  account  terrain  clobber,  survivability,  and 
navigational  reference  locations.  But  these 
algorithms  are  also  computationally  intensive  and 
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will  require  specialized  chips  to  run  real  time 
interactive  (Ref  6). 

The  last  step  that  must  be  coupled  to  the  route  plan  is 
to  produce  a  viable  terminal  area  plan  that  maximizes 
the  potential  to  kill  a  target  while  balancing 
survivability  and  the  ability  to  verify  the  kill.  I 
would  like  feedback  on  efforts  in  this  area, 
particularly  efforts  that  consider  uncertainty  in 
registration  of  the  aircraft  to  the  target,  aircraft 
kinematic  limitations,  weapon  characteristics, 
sensor  characteristics,  target  characteristics  and 
vulnerabilities,  mission  assessment,  and  "smart 
search.” 

Contingency  planning,  as  illustrated  in  Fig  6,  will 
grow  in  importance  as  capability  and  demand  for 
dynamic  reallocation  of  aircraft  increases.  The 
mission  prepared  for  may  not  be  the  mission  flown. 
To  enhance  both  the  command's  ability  to  reallocate 
on  the  fly  and  the  success  of  the  mission,  the 
planning  should  consider  operations  being  conducted 
adjacent  to  the  mission  area.  Given  the  aircraft's 
configuration,  weapons,  etc.,  it  is  possible  to 
evaluate  suitability  for  contingencies  that  could  arise 
and  to  plan  for  those  where  a  capability  exists. 

Rapid  mission  assessment  is  the  single  most 
important  aspect  of  mission  planning.  If  an 
unverified  kill  requires  another  mission,  then  the 
approach  to  maximizing  probability  of  kill  and 
aircraft  survivability  is  to  plan  for  and  to  be  able  to 
rapidly  verify  that  kill.  Best  estimates  of  weapon 
impact  points,  along  with  supporting  data  such  as 
targeting  sensor  video,  designator  aimpoints, 
timelines  verifying  designator  maintenance  through 
weapon  impact,  etc.,  must  be  collected  in  the  aircraft 
and  automatically  registerable  with  the  mission 
planning  data  base,  as  illustrated  in  Fig  7. 

Mission  control  and  rehearsal  package  preparation 
and  mission  rehearsal  are  separated  because  of  the 
focus  on  response  time  and  dynamic  allocation  of 
aircraft.  The  mission  planning  output,  listed  in  Fig 
8,  must  include  everything  necessary  for  mission 
accomplishment  and  for  the  mission  rehearsal.  If  the 
time  and  facilities  exist  for  rehearsal,  such  as  the 
virtual  display  systems  being  demonstrated  within 
the  military  (Ref  7)  and  now  available  for  multiplayer 
video  game  participants,  the  pilot  must  be  permitted 
an  avenue  to  modify  the  plan  within  the  operational 
constraints. 

5.  AIRCRAFT  SUBSYSTEM 
Like  mission  planning,  future  aircraft  must  be  highly 
automated  and  efficiently  and  effectively  interfaced  to 
the  crew.  Again,  it  must  be  developed  with  the  pilot 
foremost  in  mind  and  with  the  assumption  that  all 
mission  planning  functionality  will  eventually  end 
up  in  the  aircraft.  The  emphasis  shifts  from  "rapid”  to 
real  time  or  better.  The  four  functions  defined  for  this 
notional  aircraft  system,  as  mentioned  above,  are: 
(1)  navigation  and  precision  registration;  (2)  cockpit 
mission  rehearsal,  attack  cueing  and  assessment;  (3) 


adaptive  mission  control;  and  (4)  data  representation 
and  storage. 

Navigation  and  precision  registration ,  as  illustrated 
in  Fig  9,  is  going  through  revolutionary  change  with 
the  advent  of  the  global  positioning  system  (GPS). 
For  the  first  time  the  potential  exists  for  everyone  to 
know  where  they  are,  in  absolute  earth  coordinates, 
to  a  high  degree  of  accuracy.  Coordination  of  allied 
forces  will  be  significantly  enhanced  as  long  as 
operations  proceed  as  planned,  or  communications 
are  adequate,  and  no  or  only  intermittent  jamming  is 
encountered.  However,  information  friend,  foe,  or 
neutral  (IFFN)  will  still  be  necessary  because  every 
friendly  position  cannot  be  known  in  the  dynamics 
of  battle.  Also  registration  to  targets  will  require  a 
GPS  survey  of  the  site,  or  registration  with  a  location 
from  which  the  target  can  be  mensurated.  Several 
target  relative  techniques  are  operational,  including 
terrain  comparison  navigation  (TERCOM)  and  digital 
scene  matching  area  correlation  (DSMAC).  Several 
others  are  under  development,  including  the 
application  of  differential  GPS  references  at 
locations  from  which  target  coordinates  can  be 
accurately  surveyed*;  variations  on  DSMAC, 
utilizing  FLIR  or  radar  significant  registration 
features  that  can  be  accurately  tied  to  target 
locations**;  and  combinations  of  registration 
techniques  tied  to  accurate  digital  terrain  maps, 
generally  referred  to  as  terrain  reference  navigation 
(TRN)  systems,  which  permit  covert  nap  of  the  earth 
pilotage  and  weapon  delivery. 

Some  form  of  both  differential  GPS  and 
terrain/feature  reference  navigation  (TRN)  for  target 
relative  delivery  will  probably  be  implemented 
because  of  the  compelling  weapon  delivery  accuracy 
and  aircraft  survivability  implications.  And  both  of 
these  systems  will  require  large  area  image  data  bases 
in  which  target  images  can  be  registered  and 
mensurated.  TRN  will  additionally  require  references 
and  digital  maps  aboard  the  aircraft. 

Cockpit  mission  rehearsal,  attack  cueing  and 
assessment  is  illustrated  in  Fig  10.  The  hallmark  of 
targeting  and  mission  planning  in  response  to 
critical  short  dwell  targets  will  be  that  it  has  to  be 
performed  primarily  in  the  cockpit.  Timelines  dictate 
that  the  most  expedient  options  to  prosecute  short 
dwell  targets  will  be  either  reallocation  of  aircraft 
involved  in  other  missions  or  vectoring  of  forward 
positioned  CAP  aircraft  awaiting  such  an 
assignment.  In  either  case  the  detailed  mission 
planning  cannot  commence  until  designation  of  the 
target  area.  To  ensure  strike  success  in  these 
situations,  multimedia  cockpit  presentation  will  be 
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neceuary  to  support  rehearsal,  target  cueing,  and 
attack  aaaeaament.  Audio,  three-dimensional  terrain 
display,  and  a  highlight  of  critical  instrument 
displays  could  be  used  to  rapidly  review  the  mission 
plan,  route,  funnel  features,  timelines,  etc.,  in  a  time 
compressed  fashion  under  pilot  control.  In  the 
terminal  area,  sensors  could  be  aligned  for  target 
registration,  and  cues  displayed  to  assist  with  target 
acquisition  and  weapon  release.  Following  the 
attack,  the  target  complex  (or  uncertainty  area  {dan 
view)  could  be  displayed,  with  best  estimates  of 
weapon  impact  points  and  probability  of  kill,  along 
with  supporting  data  such  as  targeting  sensor  video, 
designator  aimpoints,  etc. 

Adaptive  mission  control  (AMC),  as  illustrated  in 
Fig  11,  serves  three  functions:  (1)  reoptimizes, 
enroute,  strike  resources,  routes  and  weapon  release 
based  upon  current  conditions;  (2)  provides  expert 
advice  on  tactics;  and  (3)  provides  the  option  of 
automatic  pilotage.  If  the  mission  has  been 
preplanned,  AMC  will  update  the  plan  based  upon  the 
situation  encountered.  If  the  mission  is  a  reallocation 
or  vectored  intercept,  AMC  will  plan  from  current 
position  and  provide  the  necessary  data  for  mission 
rehearsal,  attack,  and  assessment,  using  the  same 
processing  techniques  embedded  in  the  ground  based 
mission  planner.  In  addition,  the  AMC  could  have  a 
fuzzy  logic  expert  system.  Such  systems  are  well 
suited  to  uncertain  and  probabilistic  situations  such 
as  this  (Ref  8).  Following  are  examples. 

•  General  core  knowledge  that  is  applicable  to  most 

missions 

a)  If  fuel  is  low,  replan  for  nearest  target. 

b)  If  there  is  little  terrain  advantage,  stay  high  and 
suppress  the  threat. 

c)  If  mission  variance  is  low,  go  automatic. 

•  Mission/strike  specific  knowledge  (strike  specific 

parameters) 

a)  If  weather  ceiling  is  low,  shift  to  contingency 
plan  "N". 

b)  If  primary  target  is  missing,  replan  to  hit  target 

ity* 

The  pilot  should  have  the  option  of  using  the  AMC 
as  an  auto  pilot  to  relieve  the  work  load  to  facilitate 
mission  rehearsal  and  assessment,  or  when  automated 
terminal  maneuver  has  a  significantly  greater 
probability  of  success  (such  as  a  calibrated  weapon 
release  off  a  target  relative  reference). 

Data  representation  and  storage,  as  illustrated  in  Fig 
12,  includes  both  the  mission  data  base,  loaded  from 
the  mission  planner,  and  the  mission  record.  The  data 
necessary  to  support  navigation  and  precision 
registration,  cockpit  mission  rehearsal,  attack 
cueing  and  assessment,  and  adaptive  mission  control 
are  potentially  beyond  the  capabilities  of  projected 
storage  technology.  Strategies  for  compressed  data 
representation  of  "N"  dimensional  targeting  data 


have  been  undertaken  (Ref  9),  and  substantial 
commercial  effort  is  under  way  in  image  compression 
to  facilitate  timely  transmission  on  local  area 
networks.  But  the  combination  of  data  base  size, 
manually  intensive  data  extraction  requirements,  and 
computational  intensity  of  compression  and 
decompression  techniques  will  keep  most  concepts 
presented  here  from  having  operational  utility  in 
dynamic  combat  situations,  unless  new  strategies  for 
knowledge  representation  emerge. 

6.  STRATEGY  AND  DEVELOPMENT 
APPROACH 

We  are  developing  a  strategy,  summarized  in  Fig  13, 
that  focuses  on  three  aspects  of  the  problem  (see 
Fig  1):  (1)  efficient  and  effective  crew  interfacing, 

(2)  a  correlated  and  integrated  mission  data  base,  and 

(3)  efficient  extraction  of  data  which  supports  rapid 
(re)targeting.  We  believe  that  by  focusing  on  what 
the  aircraft  and  crew  need,  how  to  best  utilize  what 
they  need,  and  bow  to  best  interface  with  the  crew  to 
provide  those  needs,  we  can  define  the  minimum 
knowledge  base  that  is  required.  Once  the  knowledge 
base  is  defined,  we  can  examine  the  current  and 
planned  data  bases  and  real  time  data  feeds,  and 
determine  how  to  represent  only  the  data  we  need.  In 
parallel,  we  can  address  efficient  extraction— data 
transformation,  registration,  sorting  and  merging, 
and  expert  control. 

Our  development  approach  has  been  to  construct  a 
strike  data  and  mission  planning  laboratory,  as 
illustrated  in  Fig  14.  The  primary  function  of  the 
laboratory  is  to  support  the  development  of  tactical 
aircraft  mission  planning  system  (TAMPS)  modules 
for  aircraft  and  weapons.  But  the  secondary  function 
is  to  provide  the  facilities  to  examine  the  issues  of  a 
correlated  and  integrated  mission  data  base,  and  to 
provide  efficient  extraction  of  data  that  supports 
rapid  (re)targeting.  Based  upon  our  current 
understanding  and  strategy,  we  are  teaming  with 
Human  Factors  experts  to  address  the  issue  of 
efficient  and  effective  crew  interfacing. 

7.  CONCLUSIONS 

There  is  a  synergistic  ground  swell  of  technical 
capability,  and  political  and  financial  pressure,  to 
vastly  improve  our  ability  to  project  force  with 
reduced  resources.  A  vision  is  forming  that  the  vast 
improvement  will  come  about  through  the  tactical 
utilization  of  all  our  reconnaissance  data  and  through 
real  time  adaptive  coordination  of  weapon  platforms. 
To  ensure  that  pilot  overload  is  not  the  weak  link  in 
the  chain,  mission  planning  and  rehearsal  and  the 
aircraft  must  be  developed  as  an  integral  system. 
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Fig  12.  Data  Representation  and  Storage. 
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•  EFFICIENT/EFFECTIVE  CREW  INTERFACING 
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Fig  14.  Strike  Data  and  Mission  Planning  Laboratory  Equipment  Configuration 
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Discussion 


QUESTION  M.  JACOBSEN 

What  time  is  required  for  a  typical  mission  planning  using  the  tools 
presented? 

REPLY 

The  tools  presented  are  notional.  Today's  capability  to  support  mission 
planning  for  a  complex  weapon  system  such  as  the  Standoff  Land  Attack 
Missile  (SLAM)  takes  several  hours,  so  you  would  probably  plan  a  mission 
the  day  before  it  is  flown.  The  planning  is  not  automated.  For  rehearsal 
of  what  the  pilot  will  see  when  the  seeker  is  turned  on,  the  pilot  is  lucky 
if  a  photograph  of  the  target  is  available. 

QUESTION  G.H.  HUNT 

What  are  the  communications  implications  of  the  on-board 
planning/replanning  concepts  which  you  have  described? 

REPLY 

The  communications  implications  are  significant.  The  potential  area 
of  response,  for  a  loitering  patrol  aircraft,  is  hundreds  of  thousands  of 
square  miles.  The  initial  data  base  requirements,  as  discussed  in  the 
paper,  are  beyond  current  storage  technologies.  The  communications  to 
support  vectoring  and  changes  to  the  data  base  could  be  prohibitive. 
Therefore  our  focus,  which  has  been  on  defining  the  minimum  information 
necessary  to  support  strike  operations,  also  minimizes  the  communications 
requirements . 

QUESTION  R.  LITTLE 

In  the  complex  scenario  which  you  have  discussed,  do  you  believe  that 
a  single-seat  aircraft  is  feasible? 

REPLY 

Not  for  many  years.  Automation  and  expert  systems  are  still  a  long  way 
from  the  required  performance  levels  to  permit  the  replacement  of  a  crewman 
with  electronics. 
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SUMMARY 

The  rapid  developments  in  avionics  and  the 
associated  processing  power  now  available 
in  aircraft  have  produced  cockpit  systems 
which  can  quickly  saturate  the  crew  with 
information. 

Only  by  understanding  man's  capabilities 
and  limitations  will  it  be  possible  to 
design  integrated  avionics  systems  which 
match  man's  requirements  and  result  in  an 
effective  man-machine  combination. 

1  INTRODUCTION 

As  successive  new  combat  aircraft  are 
introduced  into  service  the  quantity  and 
complexity  of  avionics  systems  are 
progressively  increased  but  man's 
information  processing  capacity  has 
remained  constant  at  a  few  bits  of 
information  per  second. 

Today's  combat  aircraft  systems  are 
directing  many  channels  of  information 
into  the  cockpit  but  the  pilot  is  still 
the  same  single  channel  device  that  he  has 
always  been.  It  is  now  essential  to 
"manage"  the  flow  of  information  to  the 
pilot  to  enable  him  to  be  provided  with 
the  relevant  data,  in  a  readily 
assimilable  form  at  the  appropriate  time. 
This  requires  considerable  cockpit 
automation  to  be  incorporated  into  the 
avionics  but  at  the  same  time,  there  needs 
to  be  an  understanding  of  the  requirements 
of  the  aircrew  and,  also  the  expertise 
that  can  be  provided  by  aircrew  which 
cannot,  yet,  be  easily  performed  by 
automatic  systems. 

Future  aircrew  aids1  such  as  the  U.K.'s 
Hission  Management  Aid,  the  U.S.'  Pilots 
Associate  or  France's  Co-Pilote 
Electronique  will  only  be  viable  if  the 
information  flow  between  the  avionics 
systems  and  the  aircrew  can  be  suitably 
managed.  This  will  be  achieved  by  the  use 
of  many  techniques  which  involve  Task 
Analysis,  Allocation  of  Function  to  man 
or  machine,  careful  study  of  feedback 
mechanisms  in  both  directions  across  the 
man-machine  interface  and  validation  by 
simulation  and  trials.  Many  attempts  to 
do  this  have  been  made  in  the  past  with 
limited  success.  Somehow,  the  aircrew 
have  made  up  for  shortcomings  in  the 
avionics  integration,  but  recent  lessons 
learnt  from  the  Gulf  War  and  elsewhere 
have  shown  that  the  piecemeal  approach  to 
avionics  integration  has  sometimes  led  to 
failure  of  the  man/machine  system  and 
aircraft  and  aircrew  have  been  lost. 


Only  by  paying  great  attention  to 
management  of  the  information  flow  between 
aircrew  and  avionics  systems  will  it  be 
possible  to  optimise  the  man-machine 
system  in  future  combat  aircraft. 

Examples  of  past  problems  and  current 
developments  in  the  management  of  data 
flow  in  the  cockpit  are  given. 

MAN  MACHINE  ATTRIBUTES 

Not  only  have  the  number  of  systems  in 
aircraft  been  rising  over  the  years,  (See 
Fig  1.)  but  the  complexity  of  the 
individual  systems  has  also  been 
increasing.  To  offset  these  trends  and  to 
attempt  to  reduce  the  correspondingly  high 
crew  workloads  in  current  combat  aircraft, 
an  increasing  use  is  made  of  automation. 
However,  just  as  most  systems  appear  to 
have  been  developed  separately  and 
integrated  at  a  very  late  stage  in  the 
development  cycle,  the  application  of 
automation  often  seems  to  have  been 
applied  in  a  random  way  and  not  always 
planned  as  an  integral  component  of  the 
-n-machine  system. 


Fig  t.  Growth  In  Combat  Aircraft  Systems  with  Year  In  Service 


Rarely  are  the  relative  merits  of  the  man 
and  the  machine  compared  to  indicate  which 
tasks  should  be  allocated  to  the  man  and 
which  tasks  to  the  machine  or  automation2. 
Fitts,  as  long  ago  as  the  1950s,  listed  a 
number  of  qualities  which  are  performed 
best  by  man  or  machine  (See  Table  1.)  yet 
little  of  his  philosophy  appears  to  be 
implemented  in  military  cockpits  as  the 
21st  century  is  approached.  For  example, 
only  a  few  years  ago,  it  was  found  in  a 
new  aircraft  weapons  system  that  the  only 
links  between  the  radar,  the  navigation 


24-2 


TABLE  1 
FITTS’  LIST 


[Quality 

Machine 

Man  j 

|  Computation 

Accurate,  fast.  Poor  error 
correction. 

Slow,  Inaccurate  but  good  at  error  correction.  [ 

Complex  activities 

Multi-channel 

Single  channel.  Time  sharing.  | 

Consistency 

Precise  repetition 

Unreliable.  Needs  monitoring.  | 

Intelligence 

Limited  at  present 

Can  deal  with  the  unpredicted  and  can 
anticipate 

J  Manipulation 

Good  at  specific  tasks. 

Great  versatility. 

|  Memory 

Good  literal  reproduction. 

Large  store,  multiple  access.  Best  for 
principles  &  strategies. 

Overload 

Sudden  breakdown. 

"Graceful  degradation”. 

Power 

Wide  range.  Consistent. 

0.2  h.p.  for  continuous  work.  Variable. 

Reasoning 

Good  deductive. 

Good  inductive. 

Speed 

Wide  range.  Consistent. 

1  second  lag. 

Sensory  Input 

Wide  range,  some  outside 
human  senses. 

Can  be  made  Insensitive  to 
other  stimuli. 

Wide  energy  range  and  variety  dealt  with  by 
one  sensor,  e.g.  eye  deals  with  location, 
movement,  colour.  Good  pattern  recognition. 
Good  detection  in  noise. 

Upset  by  environmental  factors  such  as  noise, 
vibration  etc. 

and  the  weapons  system  were  through  the 
man5.  To  estimate  the  speed  and  direction 
of  a  surface  vessel,  the  crew  member  had 
to  place  cross  wires  over  the  blip  on  the 
radar  screen  and  then  read  off  a  range  and 
bearing.  This  was  fed  manually  by  keying 
into  the  aircraft's  navigation  system 
which  converted  the  data  into  a  grid 
position, read  off  and  then  plotted 
manually  onto  a  chart.  This  procedure  was 
repeated  a  minute  later.  Speed  and 
direction  were  then  estimated  from  the 
distance  between  the  two  spots  marked  on 
the  chart.  This  was  an  extremely  tedious, 
inaccurate  and  time  consuming  process  for 
the  man  which  prevented  him  from 
performing  more  important  mission 
management  tasks.  Thus,  it  was  the  man 
who  provided  the  system  integration  even 
though  the  radar  and  navigation  systems 
could  have  been  hard  wired  together 
relatively  simply  and  cheaply.  This  is  a 
typical  example  of  how  different  systems 
can  be  developed  separately  with  little 
consideration  given  either  to  their 
integration  or  to  making  the  best  use  of 
the  man's  unique  decision  making 
abilities. 

Even  when  a  decision  has  been  taken  to 
allocate  some  functions  to  be  done 
automatically,  a  further  decision  needs  to 
be  taken  over  whether  the  man  should  be 
given  the  opportunity  to  intervene  or 
merely  monitor  the  automatic  process. 

On  the  one  hand,  if  the  man  decides  to 
take  over  a  task  which  is  normally  done 


automatically  and  with  which  he  is  out  of 
practice,  he  may  well  reduce  performance 
or  exacerbate  an  already  dangerous 
situation.  Alternatively,  a  situation 
which  was  not  envisaged  when  the 
automation  was  designed  may  be  encountered 
for  which  automation  cannot  cope,  yet  the 
man,  through  inductive  thinking  can  find  a 
solution.  For  example,  some  years  ago  a 
three-engined  jet  airliner’s  all-moving 
tailplane  mechanism  failed  and  the  auto 
pilot  was  left  with  no  attitude  control. 

By  application  of  their  past  experience 
the  crew  were  able  to  save  the  situation 
by  using  differential  thrust  between  the 
rear  fin  mounted  engine  and  the  underwing 
engines  to  maintain  the  required  attitude 
and  to  land  safely.  There  is  no  way  that 
automation  could  be  planned  to  correct 
such  an  unusual  and  unexpected  situation. 

2  Transmission  Rates  and  Information 
Load 

There  is  almost  no  limit  to  the 
information  transmission  rate  for  which  a 
machine  can  be  designed,  yet  man's 
evolutionary  process  has  resulted  in  him 
having  a  transmission  rate  of  only  a  few 
bits/second.  Fig  2.  contrasts  the 
unchanging  low  information  transmission 
rates  for  man  with  the  increasingly  higher 
rates  that  are  being  developed  for 
machines.  Man's  useable  channel  capacity, 
depending  on  the  method  of  encoding  used, 
is  generally  between  about  2-25 
bits/secpnd,  regardless  of  the  channel 
employed  .  This  results  in  an  obvious  mis- 


Fig  2.  Comparison  of  Man  and  Machine  Information 
Transmission  Rates  with  System  Evolution 


natch  between  the  nan  and  the  modern 
cockpit  system's  information  handling 
capacities  of  many  millions  of 
bits/ second. 


In  addition  to  the  aircraft's  basic 
systems  such  as  flight  control,  fuel, 
weapons  etc,  the  next  generation  of 
aircraft  could  have  a  suite  of  sensors  and 
communications  links.  These  could  include 
Radar,  Radar  Warning  Receivers,  Infra  Red 
Search  and  Track,  Missile  Approach  Warner, 
Radio  Altimeter,  Terrain  Following  System, 
Cable  Warning,  Laser  Warning  Receiver, 
Night  Vision  Aids,  Data  Links,  UHF  and  VHF 
radios.  While  the  crew  can  cope  with  one 
or  two  sensors  and  systems  at  a  time,  some 
form  of  machine  intelligence  in  the  form 
of  a  mission  management  aid  will  be 
required  to  filter  the  information  and  to 
schedule  it  in  a  timely  and  appropriate 
way  to  the  crew. 


Fig  3.  Man’s  Single  Channel  for  Sensory  Inputs 


Although  man  can  be  considered  as  a  multi¬ 
sensor  device,  the  link  between  his 
sensors  and  the  "central  processor"  part 
of  his  brain  can  be  likened  to  a  simple 
channel  which  accepts  information  from 
only  one  sensor  at  a  time.  This  concept 
is  depicted  in  Fig  3.  If  it  is  assumed 
that  the  human  operator  receives  and 
processes  information  in  this  way,  it  can 
be  argued  that  the  information  from 
different  senses  must  be  time  shared. 

This,  together  with  the  limited 
transmission  capacity  re  enforces  the 
theme  of  this  paper  that  too  much 
information  should  not  be  presented  to  the 
man. 

If  the  signals  reaching  his  sensors  are 
degraded  by  noise  or  other  environmental 
effects,  his  workload  will  be  increased 
and  his  ability  to  process  the  information 
presented  will  be  still  further  reduced. 
Fig  4.  depicts  both  the  information 
sources  and  some  of  the  factors  which 
degrade  man's  ability  to  sense  the  process 
signals.  Only  by  carefully  matching 
information  sources  to  the  man's 
processing  and  channel  capacities  will  the 
optimum  man-machine  system  be  produced. 


Task  Difficulty 

Fig  4.  Contributors  to  Crew  Workload 


One  method  which  attempts  to  overcome  the 
mis-match  in  information  flow  between 
machine  and  man  is  that  of  increasing  the 
size  of  the  crew. 

For  example,  long  range  maritime  patrol 
aircraft  often  have  a  crew  of  10  or  more 
to  operate  and  integrate  the  many  systems 
required  for  Air  Sea  Warfare  (ASW) .  An 
ASW  aircraft  crew  typically  comprises: - 

2  Pilots  for  flight  control  and  systems 
management 

2  Navigators  for  routine  and  tactical 
navigation 

3  Underwater  Sensor  Operators 
1  Radar  Operator 

1  Electronic  Support  Measures  Operator 
1  Magnetic  Anomaly  Detection 
Operator/Despatcher 
Total  10 
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Even  with  this  crew  size,  there  are  often 
workload  peaks  which  are  beyond  the  crew's 
capacity  to  handle  effectively.  One 
solution  that  is  often  applied  to  overcome 
this  problem  is  to  add  one  or  more 
additional  crewmen.  This  tends  to  be  an 
ineffective  and  costly  remedy  since  it 
exacerbates  one  of  the  original  causes  of 
the  problem,  which  is  that  of 
communication  overhead. 

With  a  multi-crew  aircraft,  an  appreciable 
amount  of  effort  is  spent  in  communicating 
between  crew  members.  The  proportion  of 
this  overhead  increases  as  the  crew  number 
increases  in  the  ratio  of 

n2-n 

2 


where  n  is  the  number  of  crew. 


Communication 
Overhead 
(3  units) 


Fig  5a.  The  Effect  of  Crew  Number  on  Communication  Overhead. 

This  is  illustrated  in  Fig  5a.  With  a  crew 
of  2,  some  effort  is  required  for 
communication  between  them.  With  a  crew 
of  3,  the  communications  is  trebled  and 
with  a  crew  of  4,  6  times  the  overhead  is 
incurred,  ten  times  with  5  crew,  fifteen 
times  with  6  crew  and  so  on.  Thus  as  the 
number  of  crew  increases,  the  proportion 
of  the  communications  overhead  increases 
at  a  greater  rate.  This  is,  of  course  a 
very  simplistic  model  which  does  not  take 
account  of  the  following  two  opposite 
effects.  Firstly,  not  all  crew  members 
may  need  to  communicate  regularly  with  all 
of  the  others. 

Crmv  of  10,  whom  not  «R  crow  communktete  with  ovofyooe  st»a 
{Communication  Ovarhaad  it  unite) 


Secondly,  in  long, 

relatively  narrow  fuselages,  as  additional 
crew  members  are  added  they  and  their 
associated  equipment  have  to  be  positioned 
further  and  further  apart  along  the  length 
of  the  aircraft  which  increases  the 
difficulty  in  communication.  (See  Fig  5b.) 
Conversely,  if  the  number  of  crew  can  be 
reduced  by  automating  some  of  the  tasks, 
the  communications  overhead  will  be 
reduced  and  the  workload  will  also 
diminish. 

3  FUNCTIONAL  ANALYSIS  OF  A  MISSION 

A  clearer  understanding  of  the  information 
flow  within  the  cockpit  and  an 
appreciation  of  both  the  man's  and  the 
machine's  merits  and  shortcomings  is 
required  if  anything  near  the  optimal 
solution  is  to  be  achieved. 


Fig  6.  Fast  Jet  Cockpit  of  1 960’s  with 
Electromechanical  Instruments 


In  the  past,  cockpits  were  full  of 
electromechanical  instruments  and  each 
instrument  was  dedicated  to  a  specific 
system.  Fig  6.  shows  a  typical  fast  jet 
cockpit  of  the  early  1960s  where  the  crew 
had  fewer  systems  to  monitor;  where  mental 
and  not  physical  action  was  needed  to 
select  information.  Contrast  this  with 
Fig  7.  which  shows  a  typical  modern  fast 
Jet  cockpit  layout. 


Fig  7.  Typical  'Glass*  Cockpit  of  Today 
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This  is  dominated  by- 
just  3  multi- function  CRT  displays,  a  HUD, 
and  a  few  standby  electro-mechanical 
displays.  It  is  a  much  simpler  and  neater 
cockpit  layout.  It  has,  however,  the 
potential  of  presenting  orders  more 
information  to  the  crew  during  a  mission. 
To  ensure  that  the  correct  information  is 
presented  at  the  right  time,  either  the 
pilot  has  to  select  the  information 
himself  or  have  the  information  selected 
automatically  for  him  by  the  system.  In  a 
"real  life"  dynamic  situation  a  mix  of  the 
two  approaches  probably  will  be  required. 
To  achieve  the  right  balance  in  this  area 
a  functional  analysis  of  the  mission  is 
required.  This  will  successively  break 
down  the  mission  into  smaller  and,  more 
detailed  items.  For  example,  for  a  combat 
aircraft  the  mission  will  break  down  as 
shown  in  Table  2. 


TABLE  2 


Air  !o  Ground 
Alt 


H  jWtiton 


Attpk 


Raconnbaanca 


HoSSi  Pra  Launch" 


Attack 


1 


EgrfM 


Actions 


Continuous 


-MaMudi 

Aaaaaa  a/c  pragma 

_  Sal  tpvad 

_  Aaaaaa  thraata 

.  Navigate  to  IP 

Counter  threats 

.  Obtain  attack 

_  Maintain  comma 

*WM 

.Check  navigation 

.  Arm  wwpont 
_  Chock  fuel 

Maintain  altitude 

Ifalnlal  ■ - - - -a 

Munmn  ipvn 

Maintain  terrain 


Monitor  tin  sels 
Monitor  fuel 
Monitor  weapons 
Monitor  otfier  systems 
Maintain  data  Ink 


First  of  all  the  type  of  mission  is 
selected,  in  this  case  an  Air-to-Ground 
mission  has  been  broken  down  into  the 
phases  of  Hostile  Ingress,  Pre-Launch, 
Attack  etc.  Each  phase  is  then  sub¬ 
divided  into  Immediate,  Intermediate  and 
Continuous  Actions.  These  are  further 
separated  into  functions  such  as  "Set 
altitude  and  Speed,"  "Assess  threats"  etc. 
These  functions  can  be  examined  in  further 
detail  to  identify  objectives  and  tasks. 
For  example,  to  set  Ingress  Altitude  and 
Speed,  the  aim  must  be  to  proceed  as 
swiftly  as  possible  (since  the  enemy  may 
have  been  alerted) .  Throttle  settings 
will  have  to  be  reset  and  altitude  altered 
depending  on  information  received  by  data 
link  or  other  sources. 

Thus,  this  process  of  task  analysis 
progressively  decomposes  a  defined  mission 
into  phases,  segments  and  tasks.  It 
allows  a  clearer  understanding  to  be  made 
of  the  component  parts  of  the  mission  at  a 
very  detailed  level.  It  enables  the 

REFERENCES 


information  available  from  the  system  to 
be  compared  with  that  required  by  the  crew 
at  a  specific  time  and  any  mismatch  in 
desired  information  flow  to  be  identified. 
It  then  allows  task  synthesis  to  be  used 
to  rebuild  a  mission  for  a  future  system 
in  which  tasks  can  be  re-allocated  to  man 
or  machine,  depending  on  their  relative 
attributes. 

4  RULES  FOR  THE  MAN  IN  THE  MAN/MACHINE 
SYSTEM 

Remember  that  man  is  effectively  a  single 
channel  device  which  must  not  be 
overloaded.  Therefore: 

1  Do  not  overload  the  man  by 

a.  giving  him  too 
much  information 

b.  giving  him 
unnecessary 
information 

c.  giving  him 
calculations  to 
do  that 

can  be  done  faster  and  more 
accurately  by  a  machine. 

2  Do  not  give  the  man  uninteresting 
repetitive  tasks., 

3  Do  not  give  him  monitoring  tasks 
where  he  is  unalerted  but  looking  for 
rare  events. 

4  Where  possible,  give  the  man  tasks 
that  he  enjoys  doing  and  is  good  at. 
Give  him  enough  to  occupy  him  but 
still  leave  him  with  enough  spare 
capacity  to  deal  with  unexpected 
events . 

5  Use  the  man's  inductive  thinking 
ability  to  deal  with  unplanned  events 
which  cannot  be  automated. 


5  CONCLUSIONS 

Man  is  relatively  poor  at  handling 
information  and  is  easily  overloaded. 
Information  from  systems  and  sensors  needs 
to  be  processed,  filtered  and  presented  to 
him  at  the  appropriate  time.  Some  form  of 
mission  management  aid  which  automates  the 
functions  that  he  is  poor  at  doing  is 
required.  Despite  man's  limitations,  he 
has  some  attributes  which  cannot  yet  be 
reproduced  by  machine  intelligence.  It  is 
essential,  therefore,  to  allocate  the 
various  component  mission  functions  to 
either  the  man  or  machine,  depending  upon 
which  has  the  appropriate  attributes. 
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Sunmry. 

With  the  advances  In  display  technology  and  the 
Increasing  use  of  software  control  more 
Information  In  the  modern  conmerclal  aircraft 
cockpit  Is  available  on  request  only  Instead  of 
continually.  Furthermore,  the  reduction  In  crew 
size  has  resulted  in  a  reduction  In  the  routine 
monitoring  of  system  parameters.  However,  advances 
In  sensory  capability  are  enabling  far  more  system 
parameters  to  be  measured.  The  combination  of 
these  trends  leads  to  the  perception  by  the 
aircrew  of  an  abrupt  transition  from  normal 
operation  to  the  need  to  deal  with  system 
malfunction.  This  paper  outlines  some  work 
undertaken  to  maximise  the  use  of  available 
information  In  order  to  maintain  the  aircrew's 
awareness  of  the  status  of  the  aircraft's  systems 
and  to  provide  advice  In  the  advent  of  malfunction 
or  abnormality.  The  appropriate  carrier  of  such 
Information  Is  the  modern  centralised  cockpit 
warning  system. 


Introduction. 

With  the  advent  of  CRT-based  software  controlled 
display  technology  there  has  been  a  decline  In  the 
number  of  displays  fitted  to  civil  aircraft  over 
the  last  ten  years.  Consequently  there  has  been  an 
Increase  In  the  amount  of  hidden  Information 
which,  while  available  on  request.  Is  not 
automatically  displayed.  This  loss  of  Information 
has  been  compounded  by  the  reduction  In  crew 
numbers  from  three  to  two  l.e.  the  loss 
of  the  flight  engineer,  resulting  In  a  reduction 
In  the  routine  monitoring  of  system  status  In  these 
aircraft.  At  the  same  time  there  have  been 
advances  in  sensor  technology.  In  particular  the 
development  of  optical  sensors,  which  have 
provided  the  ability  to  measure  system  parameters 
with  greater  accuracy  and  system  parameters  which 
previously  could  not  be  measured  because  of 
hostile  environmental  conditions. 

The  result  of  these  advances  is  that,  while  there 
is  increased  measurement  of  system  parameters 
there  Is  less  capability  in  the  cockpit  to  monitor 
system  status  and  trends.  As  a  consequence,  when 
an  emergency  does  occur  the  aircrew  are  less 
prepared  to  respond  In  the  correct  manner.  This  Is 
compounded  by  other  advances  In  cockpit 
technology,  e.g.  flight  management  systems,  which 
have  reduced  crew  workload  but  potentially  also 
reduced  crew  awareness  of  the  aircraft's  state. 

As  part  of  the  process  of  reducing  crew  workload 
aircraft  warnings  and  cautions  have  been 
integrated  into  a  central  warning  system  which  is 
usually  combined  with  a  display  of  primary  engine 
parameters.  The  purpose  of  these  systems  is  to 
locate  all  aircraft  system  Information  In  a 
central  position;  audible  warnings  direct  the 
crew's  attention  to  the  central  warning  system 
which  indicates  the  problem  and  re-directs 
attention  to  the  specific  system. 


This  paper  describes  some  aspects  of  work 
undertaken  towa^  >  the  proof-of-concept  of 
an  Advanced  Cocx^it  Warning  System  (ACWS). 

This  system  Is  based  upon  making  better  use  of  the 
information  available  by  monitoring  aircraft 
systems'  status  and  trends  and  integrating  the 
results  to  achieve  not  only  local  system 
monitoring  but,  more  importantly,  a  view  of  the 
aircraft's  situation  as  a  whole.  There  are  clearly 
two  distinct  but  related  parts  to  this  work, 
namely  the  processing  of  data  and  the  display  of 
information.  This  paper  is  concerned  with  the  data 
processing  aspects  of  the  system,  the  HCI  aspects 
are  addressed  In  [1]. 


State  of  tbe  Art 

Although  there  are  minor  differences  between 
aircraft  types  the  basic  layout  of  modern  central 
warning  systems  Is  similar.  They  consist  of  two 
CRT  screens  set  one  below  the  other;  the  upper 
carries  the  primary  engine  data  together  with  an 
area  for  the  display  of  colour  coded  warnings;  the 
lower  carries,  upon  selection,  either  an  extension 
of  the  upper  display  or  a  synoptic  diagram  of  one 
of  a  set  of  aircraft  systems  e.g.  hydraulics, 
brakes  etc. 

The  colour  codelng  of  the  warning  messages  Is 
typically  red  for  Immediate  action,  amber  for 
action  to  be  taken  In  the  near  future  and  some 
other  colour,  often  magenta,  for  advisory 
information.  Indentation  may  also  be  used  to 
indicate  different  degrees  of  urgency  within 
the  amber  band.  The  warnings  are  prioritised 
according  to  this  colour  codeing  and, 
within  each  code,  according  to  the  time  of 
receipt. 

When  a  red  warning  is  received  immediate  action 
must  be  taken;  there  is  no  time  to  supply 
additional  Information.  On  the  other  hand,  when  an 
amber  warning  or  advisory  information  is  received 
there  is  a  time-window  of  undetermined  duration 
which  can  be  used  for  the  provision  of  additional 
information  and  the  delay  of  action.  Informal 
discussion  with  aircrew  has  indicated  that  it 
would  be  desirable  to  use  this  opportunity  to 
provide  information  about  the  consequences  of 
taking  the  appropriate  action  before  it  is  taken 
l.e.  what  systems  will  be  lost  when  the  action  is 
taken. 

Once  a  warning  message  has  been  received  the 
action  or  actions  to  be  taken  are  set  out  under 
standard  operating  procedures.  These  are  defined 
by  the  Flight  Operation  Manual  and  the  associated 
Quick  Reference  Handbook.  These  manuals,  to  which 
the  aircrew  refer  in  the  event  of  a  malfunction  or 
abnormal  behaviour,  are  designed  by  the  aircraft 
manufacturer  and  the  airline  and  have  been 
certified  by  the  appropriate  authorities.  Also  the 
aircrew  have  been  trained  to  respond  to  situations 
according  to  this  set  of  rules.  There  is  therefore 
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a  well-defined  procedure  to  be  carried  out  for 
each  malfunction. 

The  codeing  and  prioritisation  method  outlined 
above  is  adequate  for  single  warnings  or  simple 
situations  in  which  several  warnings  appear. 
Difficulties  can  arise  though  in  complex 
situations  where  a  number  of  failures  have 
occured  (multiple-source/multiple  effect)  or  when 
a  failure  has  resulted  in  a  number  of  consequent 
failures  malfunctions  or  abnormalities 
(s ingle-source/mu Itiple  effect).  Under  these 
conditions  the  number  of  warnings  and  advisories 
combined  with  the  pressure  to  respond  and  the 
rapid  transition  from  a  normal  to  an  abnormal 
situation,  can  cause  confusion  leading  to 
Inappropriate  actions  being  taken.  In  the  case  of 
multiple-source  multiple  effect  failures  the 
difficulties  are  compounded.  Not  only  is  too  much 
information  supplied  but  also,  once  the  causes 
have  been  ascertained  it  is  necessary  to  devise  a 
set  of  actions  which  will  allow  all  failures  to  be 
rectified  i.e.  actions  taken  to  rectify  one 
failure  must  not  inhibit  remedial  action  being 
taken  for  another. 

The  requirements  for  an  effective  ACWS  are  derived 
from  the  above  discussion.  It  is  intended  that  the 
resulting  system  performs  the  following  functions: 

-  long-term  monitoring  of  the  type  previously 
undertaken  by  the  flight  engineer; 

-  the  identification  of  primary  causes  of 
abnormal  conditions  paricularly  in  cases  of 
multiple  effects: 

-  the  provision  of  remedial  actions,  and  their 
consequences; 

-  a  reduction  in  the  occurence  of  false 
warnings; 

-  maximisation  of  the  time  between  the 
detection  of  abnormalities  and  the  occurence 
of  emergencies. 


System  Overview. 

The  system  being  investigated  is  based  upon  the 
integration  of  a  set  of  real-time  monitoring 
systems;  each  such  system  is  responsible  for  its 
local  monitoring  and  an  executive  function 
integrates  the  output  of  the  set.  As  such  it  has 
three  principle  components  (see  Fig.l). 

At  the  lowest  level,  receiving  data  from  the 
aircraft  systems,  is  a  network  of  nodes.  Each  node 
is  a  monitoring  system  responsible  for  assessing 
the  status  and  trends  of  the  output  parameters  of 
the  aircraft  system  which  it  represents.  To  do 
this  each  node  contains  a  model  of  the  system  it 
is  monitoring  together  with  an  assessment  function 
which  is  able  to  report  on  the  status  and  trend  of 
the  output  parameters  of  the  real-world  system. 

The  nodes  are  linked  to  each  other  in  a  manner 
which  represents  the  transfer  of  physical 
parameters  from  one  aircraft  system  to  another. 

The  nodes  report  system  status  to  an  executive 
function,  the  network  executive,  which  combines 
the  Information  it  is  receiving  to  form  a  view  of 
the  overall  situation.  From  this  view  the 
executive  forms  a  plan,  an  ordered  set  of  actions, 
derived  from  standardoperating  procedures,  to 
ameliorate  abnormalities,  rectify  the  effects  of 
malfunctions  and  maintain  the  safe  operation  of 
the  aircraft. 


The  third  component  of  the  system  is  the  display. 
Work  is  being  undertaken  [1]  in  this  area  to 
ascertain  the  difficulties  encountered  with  the 
current  display  layout  and  the  difficulties  that 
will  be  encountered  when  additional  information  of 
the  type  produced  by  the  proposed  system  is 
available. 


System  Operation. 

As  each  node  monitors  its  associated  real-world 
system  it  uses  its  model  to  take  inputs  from  its 
connected  nodes  and  produce  values  of  its  output 
parameters.  Each  of  these  is  compared  to  the 
sensed  values  of  the  parameters  and,  if  a 
discrepency  is  detected,  a  report  containing 
status  and  trend  information  is  passed  to  the 
executive  function.  Both  the  sensed  parameter 
values  and  the  reports  are  passed  on  to  the 
connected  nodes.  In  this  way  a  node  is  aware  of 
the  quality  of  its  inputs  and  any  detection  of 
fault  can  automatically  be  ascribed  to  the  system 
being  monitored.  In  this  manner  two  of  the  ACWS 
requirements  are  met;  by  contiuously  monitoring 
individual  parameters  long-term  effects  such  as 
a  slow  oil  leak  can  be  detected  sufficiently  early 
for  appropriate  action  to  be  taken;  by 
interlinking  the  monitoring  systems  the 
distinction  between  cause  and  effect  can  be 
established,  thereby  resolving  the  confusion 
resulting  from  the  over-provision  of  information. 

In  the  case  of  a  single-source  malfunction  the 
executive  is  required  to  look  up  the  appropriate 
actions,  derived  from  the  standard  operating 
procedures  and  pass  them  on  to  the  display.  Before 
doing  so  it  may  also  check  the  report  by  searching 
for  confirmatory  data  among  the  other  reports  it 
has  received.  This  will  have  the  effect  of 
reducing  the  number  of  false  warnings.  In  the  more 
complex  cases  of  multiple-source  malfunctions  the 
executive  must  order  the  set  of  actions  which  is 
composed  of  all  actions  for  each  malfunction  as 
derived  from  the  standard  operating  procedures.  In 
each  case  the  executive  can  use  the  network  of 
nodes  as  a  resource  to  investigate  the 
consequences  of  particular  actions  or  orderings  of 
actions.  In  this  manner  other  system  requirements 
are  fulfilled  i.e.  prediction  of  the  consequences 
of  taking  an  action  before  it  is  taken. 


Model-Based  Reasoning. 

The  use  of  models  to  represent  and  understand 
situations,  events  and  systems  has  been  a  key 
feature  of  many  diagnostic  systems  developed  in 
the  field  of  Artificial  Intelligence  (AI).  The 
model -based  approach  to  reasoning  can  be  seen  as 
the  basis  of  many  comnon-sense  reasoning  tasks. 

Models  can  be  used  in  two  distinct  ways.  The  first 
is  as  a  tool  or  method  for  containing  our 
understanding  of  the  real  world.  In  this  case 
observations  which  do  not  accord  with  the 
predictions  of  the  model  are  assumed  to  require  a 
revision  of  the  model.  This  is  essentially  the 
mechanism  by  which  we  attempt  to  understand  the 
natural  world  through  science,  improving  our 
understanding  through  observation  and  refinement 
of  our  models. 

In  the  second  case  the  model  is  assumed  correct 
and  observations  of  a  real-world  system  which  do 
not  accord  with  the  model  are  assumed  to  be 
symptoms  of  a  malfunctioning  of  the  corresponding 
real-world  system.  This  is  essentially  the 
diagnostic  use  of  the  model.  Within  this  category 
a  number  of  systems  have  been  developed.  There 


are  those  in  which  models  represent  the  symptoms 
of  a  malfunctioning  system;  medical 
diagnostic  systems  such  as  MYCIN  [2].  Such  systems 
contain  only  known  symptoms  derived  from  human 
expertise.  In  consequence  they  are  clearly 
unable  to  diagnose  any  malfunction  which  has  not 
already  been  recognised.  This  limitation  extends 
to  being  unable  to  perform  a  diagnosis  when  the 
symptoms  of  more  than  one  malfunction  are  observed 
simultaneously.  These  models  lack  the  "deep 
knowledge"  to  allow  reasoning  from  first 
principles. 

As  a  result  of  the  limitations  of  these  expert 
systems  interest  has  recently  turned  to  diagnosis 
from  first  principles  using  models  which  represent 
the  behaviour  of  the  correctly  functioning 
real-world  system  [3],  [4],  and  [5]  for  example. 

In  systems  such  as  these  observations  which 
conflict  with  the  model  cause  diagnosis  to  be 
undertaken  usually  through  the  use  of  a  heirarchy 
of  models.  While  there  are  no  discrepencies  at  the 
top  level  only  the  top  level  model  is  used;  when  a 
discrepency  is  found  at  that  level  the  next  lower 
level  models  are  checked.  This  process  continues 
until  a  model  at  the  lowest  level  is  found  to  be 
faulty  i.e.  the  cause  has  been  identified  and 
action  can  be  taken.  In  the  case  of  multiple 
faults  this  process  can  become  complex  although 
the  principle  is  the  same.  Within  such  an  approach 
it  is  necessary  to  determine  the  level  of  detail 
to  which  the  modelling  need  be  taken  and  this  in 
turn  can  be  determined  by  the  actions  which  will 
be  taken  to  rectify  the  situation.  For  example,  if 
the  remedial  action  is  to  replace  a  unit  in  a 
system  then  there  is  no  purpose  in  modelling  the 
internals  of  that  unit;  it  is  sufficient  to 
identify  which  jnit  is  faulty. 

This  approach  to  modelling  is  the  one  which  has 
been  adopted  for  the  ACWS.  Since  the  nature  of  the 
remedial  actions  which  can  be  taken  by  the  aircrew 
in  response  to  a  malfunction  are  limited  (usually 
confined  to  reducing  use  of  the  equipment  in 
question  or  turning  it  off  entirely),  the  system 
is  required  to  identify  malfunctioning  systems 
but  it  is  not  required,  at  present,  to  undertake 
thorough  diagnosis.  Consequently  there  Is  little 
purpose  in  diagnosing  to  a  level  lower  than  that 
which  is  useful  for  the  aircrew.  Furthermore  it 
can  be  assumed  that,  for  the  majority  of  its 
operation,  the  ACWS  is  confirming  that  the  systems 
it  is  monitoring  are  behaving  normally  and  it 
would  be  fruitless  to  expend  considerable 
computing  power  running  detailed  models  to 
confirm  this.  It  follows  then  that  the  models 
need  to  be  sufficient  to  detect  abnormal  behaviour 
and  place  some  measure  on  it  but  do  not  require 
the  level  of  detail  that  would  be  required  for  an 
on-board  diagnostic  system.  If,  at  later  stage  the 
ACWS  were  to  be  extended  to  an  on-board  diagnostic 
and  maintenance  system  it  is  highly  probable  that 
simple  models,  similar  to  those  currently 
employed,  would  be  used  for  detecting 
abnormalities  and  that  advice  to  the  aircrew  would 
be  given  on  the  basis  of  these  models  while  more 
complex  models  would  be  used  for  detailed 
diagnosis  which  could  be  undertaken  as  a 
background  task  once  the  safety  of  the  aircraft 
had  been  established. 


The  Use  of  Models  for  Monitoring. 

Within  the  ACWS  application  the  types  of 
parameters  to  be  monitored  can  be  broadly  divided 
into  two  classes.  There  are  those  which  should 
have  a  constant  or  almost  constant  value 
throughout  the  operating  cycle  of  the  aircraft;  an 
example  of  this  type  of  parameter  would  be  the  oil 


quantity  in  an  engine.  Monitoring  these  parameters 
requires  assessing  whether  any  changes  are 
occurring  to  the  value  and  whether  such  changes 
constitute  a  trend.  In  this  case  the  model  used 
for  monitoring  could  be  quite  simple  although  some 
complexity  is  added  when  the  start-up  and 
shut-down  phases  of  the  operating  cycle  are 
included. 

The  second  category  comprises  those  parameters 
which  are  in  some  manner  dependent  upon  the  values 
of  some  input  parameters:  in  the  engine  the 
turbine  rpm  could  be  said  to  depend  upon  the 
combustion  gas  temperature.  In  such  cases  only  the 
extremely  high  or  extremely  low  values  of  the 
parameter  constitute  abnormalities  in  their  own 
right.  Between  these  extremes  an  abnormality  is  a 
mis-match  between  the  values  of  the  input 
parameters  and  the  expected  values  of  the  output 
parameters. 

The  task  of  monitoring  aircraft  systems  within  the 
ACWS  is  based  upon  the  comparison  between 
real-world  values  of  parameters,  derived  from 
sensory  apparatus,  with  expected  values  of  those 
parameters  derived  from  a  model  or  models.  The 
value  of  a  parameter  is  described  as  either  too 
high,  too  low  or  normal.  A  detected  discrepency 
will  be  tracked  to  determine  its  trend  which  is 
categorised  as  one  of: 

convergent  -  the  situation  is  improving; 

divergent  -  the  situation  is  deteriorating; 

stable  -  the  value  is  too  high/low  but 
neither  improving  nor  deteriorating; 

oscillating  -  the  situation  is  unstable. 


Rather  than  use  computationally  intensive  detailed 
models  of  the  systems  to  be  monitored  the  approach 
used  to  meet  the  requirements  of  the  system  is  to 
base  the  model  upon  a  set  of  relationships  between 
parameters  which  are  derived  from  data  from  the 
particular  system  to  be  monitored,  and  an 
understanding  of  how  such  a  system  functions.  In 
the  case  of  an  engine  a  part  of  a  simple  model 
could  be  of  the  form: 

If  FUEL-FLOW  is  HIGH 

then  TURBINE  GAS  TEMPERATURE  is  HIGH. 

Such  a  model  is  too  simple;  it  does  not  consider 
the  rate  of  change  of  either  parameter  but  it 
could  be  extended  to  do  so.  It  can,  however, 
identify  an  abnormal  condition  if  say  Fuel  Flow 
was  seen  to  be  high  but  Turbine  Gas  Temperature 
was  detected  as  not  being  high.  This  is  a  model 
of  correct  behaviour  rather  than  a  description  of 
a  particular  failure  condition;  any  pair  of  Fuel 
Flow/Turbine  Gas  Temperature  values  which  does  not 
accord  with  this  model  could  be  described  as 
abnormal  in  the  sense  of  not  complying  with 
expected  behaviour.  Hence  normality  and, 
consequently,  abnormality  are  detected  quickly  by 
determining  the  truth  of  the  statement  for  any 
given  pair  of  values. 

In  order  to  establish  the  trend  of  any  discrepency 
it  is  necessary  to  assign  some  value  to  it.  In  the 
above  example  that  would  mean  having  some  measure 
of  the  degree  to  which  Fuel  Flow  is  HIGH  and 
Turbine  Gas  Temperature  is  HIGH.  Therefore  a 
classification  mechanism  is  required.  This 
mechanism  is  supplied  by  the  use  of  fuzzy  logic. 


An  Overview  of  Fuzzy  Logic. 

Fuzzy  sets  were  conceived  of  as  a  method  of 
representing  classes  of  objects  found  in  the  real 
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world  which  do  not  have  precisely  defined 
membership  criteria.  Upon  consideration  this  can 
be  seen  to  cover  a  wide  range  of  classes  of  object 
which  play  an  important  role  in  human  thinking 
and  decision  making  processes.  For  example,  the 
"set  of  tall  men"  is  an  adequate  definition  of  a 
class  which  can  be  reasoned  about  in  a  manner 
such  as,  "tall  men  need  to  buy  large  size 
clothes’.  It  would  be  difficult,  and  possibly 
misleading,  to  attempt  to  define  "tall  men"  and 
"large  size  clothes”  in  too  great  a  detail  and 
such  detail  is  not  necessary  for  our  understanding 
of  the  statement.  This  is  the  type  of 
categorization  required  by  the  models  in  the  ACWS 
in  order  to  perform  the  monitoring  task. 

Since  the  introduction  of  fuzzy  sets  by  Zadeh  in 
1965  [6]  the  concept  of  fuzzy  logic  has  been 
developed  and  applied  in  a  wide  range  of  fields. 
Initial  work  was  involved  in  making  more  rigorous 
the  formulation  of  the  theory  including 
constructing  common  definitions  of  the  usual  set 
operations  (see  [7],  [8]  for  example).  However,  it 
became  clear  at  an  early  stage  of  development  that 
fuzzy  set  theory  had  applications  to  control 
problems  and  a  number  of  experimental  systems  were 
developed  across  a  range  of  application  domains 
e.g.  [9], [10].  The  use  of  fuzzy  control  has  now 
reached  the  market  place  in  Japan  and  China  with 
the  Japanese  electronics  industry  replacing 
conventional  control  systems  with  fuzzy  control 
systems  in  a  wide  range  of  products.  These  include 
consumer  appliances  such  as  washing  machines, 
video  cameras  and  vacuum  cleaners  as  well  as  many 
industrial  applications  such  as  failure  prediction 
for  machine  tools,  the  control  of  automatic  trains 
[11],  the  control  of  crane  operations  [12],  and 
the  control  of  water  quality  [13], 

A  fuzzy  logic  controller  has  at  its  core  a  model 
which  is  composed  of  a  set  of  production  rules 
familiar  to  designers  of  expert  systems.  Such 
rules  are  of  the  form: 


and  the  assignment  of  5.0  is  then: 

SHORT:  0.8; 

AVERAGE:  0.2; 

TALL:  0.0 

providing  a  classification  of  this  value. 


Monitoring  within  the  ACWS. 

The  monitoring  process  in  the  ACWS  is  composed  of 
three  elements;  a  fuzzy  set  classifier,  a 
rule-based  model  and  a  comparison  function.  These 
are  illustrated  in  Fig  3. 

The  fuzzy  set  classifier  takes  the  value  of  a 
parameter  and  assigns  a  distribution  over  the  five 
sets  EXTREMELY-L0W,  LOW,  MEDIUM,  HIGH  and 
EXTREMELY-HIGH.  This  distribution  is  represented 
as  a  five-element  vector  of  the  form: 

(0,  0,  y,  1  -  y,  0) 

where  the  sum  of  the  elements  is  1.0  and 
non-zero  values  are  assigned  to  two  adjacent 
components.  Thus  a  value  of  a  parameter  derived 
from  the  sensors  is  converted  into  a  description 
of  the  value.  In  a  similar  manner  changes  in  the 
value  over  time  are  reflected  in  changes  in  the 
degree  of  membership  of  each  of  the  sets.  The 
classifier  is  unique  to  the  system  being  monitored 
and  the  shape  of  the  sets  is  derived  from  data 
from  that  system  e.g.  classification  of  turbine 
gas  temperature  may  be  slightly  different  for  each 
engine  on  the  aircraft  although  the  engines  are 
nominally  the  same. 

The  rule-based  models  are  adapted  from  those  used 
by  fuzzy  control  systems  as  described  above.  In 
this  case,  rather  than  deducing  an  output  value 
the  model  produces  an  output  distribution  such  as 

(0,  0,  z,  1  -  z,  0) 


IF  x  is  A,  and  y  is  B,  then  z  is  C, ; 


from  an  input  distribution  e.g. 


IF  x  is  Aj  and  y  is  B3  then  z  is  C2; 


(0,  0.  x,  1  -  x,  0). 


IF  x  is  A3  and  y  is  B3  then  z  is  C3. 


This  achieved  by  using  rules  of  the  form: 


A  rule  may  have  more  antecedents  and  more  complex 
logical  relations.  The  membership  of  x  in  the  sets 
A,  (where  i  -  1,  2.  or  3  in  this  example)  and  the 
membership  of  y  in  the  sets  Bj  is  determined  from 
the  use  of  fuzzy  set  membership  functions.  The 
purpose  of  the  rules  is  to  determine  an 
appropriate  value  of  z  by  inferring  its  membership 
value  in  the  sets  C|(  from  the  membership  values 
of  x  in  the  sets  A,  and  y  in  the  sets  B,. 

This  is  the  type  of  model  employed  in  the  ACWS. 

The  model  itself  is  independent  of  the  way  in 
which  values  are  distributed  over  the  sets  i.e.  it 
is  a  description  of  the  interaction  between 
parameters.  The  parameters  are  ascribed  set 
membership  on  the  basis  of  data  and  an 
understanding  of  the  system  under  consideration. 


An  example  of  assignment  of  parameter  values  to 
fuzzy  sets  is  shown  in  Fig. 2  where  the  parameter 
under  consideration  is  height  and  the  value  being 
assigned  is  5.0.  The  sets  in  Fig. 2  can  be 
described  as: 


SHORT: 

f,(h) 

-  1.0, 

h  < 

4.8, 

=  (5.8 

-  h), 

4.8 

<■  h  <- 

5.8; 

AVERAGE: 

f2(h) 

-  (h  - 

4.8), 

4.8 

<-  h  <« 

5.8, 

-  (6.8 

-  h). 

5.8 

<-  h  <- 

6.8; 

TALL: 

f3(h) 

-  (h  - 

5.8), 

5.8 

<■=  h  <= 

6.8; 

TEMPERATURE  is  LOW 

to  degree  FUEL-FLOW  LOW 

if  FUEL-FLOW  is  LOW  and  STABLE 
or  FUEL-FLOW  is  LOW  and  INCREASING 
to  degree  FUEL-FLOW  EXTREMELY-L0W 

if  FUEL-FLOW  is  LOW  and  INCREASING 
or  FUEL-FLOW  is  EXTREMELY-L0W 

etc.,  one  set  for  each  of  the  five  categories. 

The  model  describes  the  behaviour  of  a  system  and 
so  is  common  to  all  such  systems  e.g.  an  engine 
model  is  the  same  for  all  engines. 

Comparison  is  performed  by  examining  the 
difference  between  the  distribution  produced  by 
the  classifier  and  that  produced  by  the  model.  The 
contrast  between  the  distributions  is: 


(0,  0,  z  -  y,  y  -  z,  0). 


The  magnitude  of  the  difference  is  [z  -  y|  and  the 
sign  of  z  -  y  determines  whether  the  value  is 
higher  or  lower  than  expected.  Clearly  with 
| z  -  y |  close  to  0  the  parameter  is  behaving  as 
expected  and  no  further  action  need  be  taken.  If, 
however,  |z  -  y|  is  not  close  to  0  then, 
potentially,  an  abnormality  is  developing  and  the 
changes  in  |z  -  y|  are  monitored  to  determine 
whether  the  deviation  is  deteriorating,  improving 


or  stationary.  Also  the  rate  of  change  in  the 
value  will  indicate  the  speed  at  which  changes  are 
occuring. 

Propogation  through  the  network  of  models  is 
effected  by  passing  the  distribution  derived  from 
the  sensed  value  to  the  connected  nodes  for  use  in 
their  own  models.  It  is  intended  that  the  network 
can  also  be  used  to  predict  the  outcome  of 
actions.  This  can  be  achieved  by  supplying  assumed 
values  or  distributions  to  the  node  with  the  model 
supplying  the  consequent  output  to  be  passed  on  to 
its  connected  nodes.  In  this  way  the  affects  of 
actions  on  the  overall  system  can  be  determined 
before  those  actions  are  taken. 

This  division  of  the  processes  allows  one 
monitoring  mechanism  to  be  employed  on  all  models 
of  all  systems  since  parameter  values  are  mapped 
into  the  range  0  -  1  by  the  classifier.  It  also 
allows  for  substitution  of  components  since  the 
model  of  the  behaviour  will  be  unchanged  but  the 
classifier  will  be  amended  in  accordance  with  data 
derived  from  the  replacement  system. 


Progress  and  Results. 

Using  the  above  methods  a  model  of  an  engine  has 
been  produced  which  is  composed  of  three 
sub-models;  a  three-stage  compressor,  a  burner  and 
a  turbine.  These  models  monitor  six  primary 
parameters,  fuel  flow,  turbine  gas  temperature, 
rpm,  and  low,  intermediate  and  high  pressure 
spooler  speeds.  Initial  results  are  promising: 
this  simple  network  is  capable  of  detecting  and 
establishing  the  trend  of  the  onset  of  an 
exceedence  in  spooler  speed  during  take-off  some 
10  seconds  before  the  exceedence  occurs.  This  set 
of  models  Is  being  refined  and  extended  to  include 
the  monitoring  of  complimentary  parameters  e.g. 
intermediate  pressure  and  temperature 
measurements.  One  such  network  will  be  required  to 
represent  each  engine  on  an  aircraft. 

The  proof  of  concept  stage  of  the  project  has  a 
further  two  years  to  run.  It  is  intended  that  at 
the  end  of  this  stage  there  will  be  models  of  the 
engines,  the  fuel  system,  the  electrical  system, 
the  hydraulic  system  and  the  flight  control 
system.  In  parallel  with  this  activity  the  other 
aspects  of  the  ACWS,  the  executive  function  and 
the  interface,  will  be  developed. 
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Discussion 


QUESTION  R.  LITTLE 

How  can  we  certify  these  systems  to  the  high  levels  of  integrity 
required  in  civil  aviation? 

REPLY 

The  need  is  recognised,  but  not  enough  work  has  yet  been  done  in  this 
area . 

QUESTION  R.M.  TAYLOR 

You  referred  to  the  necessity  of  using  relatively  simplified  models  of 
complex  systems.  How  do  you  evaluate  and  validate  whether  the  model  is 
necessary,  sufficient  and  adequate  for  the  task? 

REPLY 

The  models  are  derived  from  available  data  and  from  a  basic 
understanding  of  the  system  being  modelled.  The  derivation  of  models  is 
essentially  data  driven;  they  are  validated  against  other  data  sets 
relating  to  the  same  system. 
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SUMMARY 


This  papar  axaainas  how  typical  oparational 
scenarios  that  ara  raprasantativa  of  futura 
conflict  impact  on  tha  spacif ication  and 
design  for  aan/aachine  performance. 
Oparational  Requirements  for  aircraft  to 
survive  and  delivar  tha  goods  in  this 
context  prasant  a  traaandous  challenge  to 
tha  Priaa  Waapon  Systaa  Contractor. 

Military  procuranant  agancias  hava  long 
baan  striving  to  raalisa  incraasad  waapon 
systaa  parforaanca  froa  dwindling 
rasourcas.  Thus  currant  initiativas  such  as 
MANPRINT  hava  baan  launchad  to  changa  an 
equipment-oriented  viaw  of  systaa 
davalopaant  towards  a  broadar  viaw  that 
considers  hardware,  software  and  operator 
together  as  a  system. 

It  is  argued  that  unless  the  piloting 
function  and  system  integration  tasks  are 
considered  as  an  integral  part  of  the 
design  process  froa  day  ona,  a  less  than 
optiaua  design  will  always  result.  It  is 
recoaaended  that  a  structured  top-down 
design  methodology  be  employed  that 
translates  Oparational  Requirements  into 
piloting  and  system  functions  at  one  and 
the  same  tiae. 

An  overview  is  given  of  use  of  just  such  a 
design  process  developed  at  BAe  Warton, 
with  particular  rafaranca  to  the  European 
Fighter  Aircraft  ( EFA)  project.  Tha  various 
stages  of  the  design  process  ara  explained 
and  indications  given  of  currant  prograss 
on  EFA.  Eaphasis  is  given  to  tha  tools  and 
aathods  used  to  ensure  that  a  highly 
integrated  systea  and  advanced  cockpit 
design  ara  successfully  achieved. 

1  IWTMOOPCTIOM 

Predicting  tha  scenario  of  the  next 
conflict  in  which  Western  aircraft  will  be 
expected  to  operate  is  becoming 
increasingly  difficult.  Tha  demands  on  the 
next  generation  of  f ightar/attack  aircraft 
ara  for  an  unpracadantad  level  of  agility, 
performance  and  lethality  combined  with 
flexibility  and  adaptability. 

This  aircraft  is  expected  to  combine  the 
roles  of  both  tha  InterDictor  Strike  (IDS) 
and  Air  Dafanca  Variant  (ADV)  of  Tornado, 
outperform  the  F15/F16,  be  more  versatile 
than  tha  FIS,  be  so  stealthy  as  to  be 
almost  invisible  whilst  remaining  a 
single-seat  aircraft.  The  ability  of  the 
pilot  to  manage  the  demanded  task  with  the 
provided  vehicle  is  therefore  a  crucial 
eleaent  in  the  design  process. 

In  many  current  military  aircraft. 


particularly  single-crew  aircraft,  pilot 
workload  levels  are  so  high  that  they  can 
limit  tha  capability  of  the  operational 
weapon  systea.  It  is  easy  to  see  how  a 
diversity  of  projected  scenarios,  hostile 
environments,  systems  capabilities  and  task 
demands  can  all  conspire  to  exacerbate  the 
workload  problem  even  further. 

The  addition  of  more  capable  systems  and 
"pilot  aids"  can  add  to  the  already 
considerable  problems  if  the  underlying 
assumption  is  that  the  pilot  can  act  as  the 
all-embracing  interface  between  all  the 
disparate  systea  elements.  As  the  pilot 
reaches  saturation  level,  he  can  become  a 
limiting  factor  in  the  effectiveness  and 
throughput  of  the  weapon  system. 

Increasing  the  level  of  automation  in 
systems  functions  has  the  potential  to  both 
enhance  the  aircraft's  capability  and  to 
ameliorate  the  pilot  workload  problem. 
Automation  of  low-level  housekeeping  tasks 
frees  the  pilot  to  concentrate  on  being 
what  he’s  best  at:  a  high-level  decision 
maker;  whilst  automation  in  the  tactical 
areas  of  identification,  analysis  and 
attack  heightens  the  performance  of  the 
man-machine  combination  in  the  very  role 
for  which  it  was  procured.  Identifying 
where  and  how  much  to  automate  is  a  crucial 
task.  One  of  the  most  important  elements  in 
this  work  is  tailoring  the  tasks  that  are 
demanded  of  the  pilot  so  that  they  are 
suited  to  his  unique  capabilities. 

It  is  thus  apparent  that  the  design  of  the 
total  man-machine  interface  and  the 
underlying  systems  as  a  coherent  entity  is 
possibly  the  most  important  task  facing  the 
prime  systems  contractor.  This  paper 
describes  a  structured  design  approach  that 
aims  to  address  these  issues  and 
illustrates  how  the  method  is  being 
employed  on  the  European  Fighter  Aircraft. 

2  STRUCTURED  TOP-DOWH  METHODS 

BAe  has  much  experience  in  the  use  of 
structured  top-down  design  methods  for  the 
analysis  and  design  of  complex  systems.  In 
particular,  two  specific  design  methods 
have  been  developed  in-house  to  facilitate 
avionic  and  crew  interface  design. 

o  Controlled  Requirements  Expression  (CORE) 
has  been  developed  as  a  tool  to  enable 
the  decomposition  of  functional 
requirements  into  detailed  systems  design 
to  be  effected  in  a  consistent  and 
unambiguous  manner.  This  approach  has 
been  used  very  effectively  on  the 
Experimental  Aircraft  Project  (EAP)  and 
is  the  basis  for  system  design  currently 
being  used  on  EFA. 
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o  In  parallel  with  CORE,  BA*  has  also 

developed  a  method  of  Mission  Analysis  to 
translat*  Operational  Requirements  into  a 
statement  of  cockpit  design  requirements . 
These  are  expressed  in  terms  of  the 
desired  level  of  systems  automation  and 
the  pilot's  need  for  information  and 
controls  within  the  cockpit. 

Althouqh  the  CORE  and  Mission  Analysis 
methods  have  been  derived  independently,  we 
now  see  benefit  in  inteqratinq  the  two 
approaches.  The  cockpit  and  systems  design 
tasks  could  now  proceed  from  a  common 
baseline  and  consistency  of  approach  can  be 
maintained  at  all  levels  of  decomposition. 

Further  details  of  the  CORE  process  as 
applied  to  systems  design  can  be  found  in 
Reference  1.  This  paper  will  concentrate  on 
the  cockpit  design  process  using  Mission 
Analysis  with  an  overview  of  the  method 
being  given  in  the  following  section. 

Although  BA*  has  been  developing  this 
analysis  technique  over  several  years,  the 
exact  form  of  its  use  on  EFA  was  derived  in 
collaboration  with  two  of  our  partner 
companies  Messerschmitt  Bolkow  Blohm  (MBB) 
and  Aerltalia  (AIT).  In  fact  development  of 
similar  methods  by  MBB  (Reference  2)  and  an 
AGARD  working  group  (Reference  3)  provides 
useful  corroboration  of  the  validity  and 
merit  of  such  an  approach. 

3  MISSIOM  AMALTSI3  METHOD  OVERVIEW 

The  structured  top-down  approach  starts 
with  an  aircraft  Operational  Requirement 
and  works  its  logical  way  down  to  a 
detailed  cockpit  design.  The  following 
interim  steps  are  generated  as  the  process 
is  pursued: 

o  A  family  of  Mission  Profiles  is  compiled; 
o  A  typical  Forcing  Mission  is  chosen; 
o  This  Forcing  Mission  is  broken  down  into 
Phases  of  Flight; 

o  All  relevant  modes  of  operation  per  phase 
are  identified; 

o  Each  phase  is  divided  into  segments; 
o  segments  are  described  by  a  Task  Listing; 
o  Functions  utilised  are  defined  per  Task; 
o  Automation  Level  is  given  per  Function; 
o  Information  and  Control  Function 

requirements  are  given  for  each  Task. 

The  identified  Functions  Utilised  and 
Automation  Level  then  feed  into  the  CORE 
process  of  detailed  systems  design.  The 
Information  and  Control  Function 
requirements  are  as  yet  implementation 
independent;  this  is  where  the  Human 
Factors  discipline  is  employed  in 
translating  the  requirements  into  a 
detailed  cockpit  layout  and  moding 
philosophies  that  undergird  the  design. 

Each  step  of  the  process  will  now  be 
described  in  more  detail. 

3.1  Forcing  Mission 

The  first  step  is  to  identify  using  the 
Operational  Requirement  a  set  of  Mission 
Profiles  that  are  representative  of  the 
uses  to  which  the  aircraft  is  expected  to 
be  put.  As  a  considerable  amount  of  effort 
would  be  required  to  break  down  every 
profile  into  a  detailed  task  listing  and 


replication  of  tasks  would  ensue,  the 
concept  of  a  Forcing  Mission  is  postulated. 

Out  of  all  the  Mission  Profiles  of  a 
particular  type  (the  Primary  Profiles  may 
be  Air  to  Air)  a  single  profile  is  selected 
as  being  forcing.  This  means  that  the  on* 
profile  is  used  to  drive  out  the  majority 
of  design  elements  that  will  also  satisfy 
the  other  profiles.  This  Forcing  Mission 
drives  the  design  process  as  a  first  pass 
to  establish  a  baseline  cockpit  layout  and 
moding  philosophy.  In  order  that  nothing 
significant  is  overlooked,  any  other  key 
profiles  can  be  given  the  same  treatment  as 
a  check  on  the  validity  of  the  cockpit 
design  that  is  taking  shape. 

3.2  Phasms  Modes  and  Segments 

Each  Mission  Profile  may  then  be  split  into 
a  series  of  mission  phases  such  as: 

o  Ground  Procedures; 
o  Take  Off; 
o  Navigation; 
o  Combat  etc. 

Each  phase  of  flight  can  be  identified  to 
have  several  modes  of  operation,  all  of 
which  pose  different  requirements  on  the 
cockpit  design.  For  instance.  Combat  may  be 
engaged  in  two  different  ways: 

o  Beyond  Visual  Range  ( BVR ) ; 
o  Within  Visual  Rang*  (WVR). 

The  BVR  option  is  concerned  with  engagement 
of  multiple  targets  at  stand-off  ranges  and 
drives  the  requirement  for  correlation  of 
information  from  all  available  sensors, 
detailed  tactical  analysis  and  planning. 

The  WVR  option  is  a  much  more  dynamic, 
rapid  reaction  task  involving  visual  target 
acquisition  and  high-G  manoeuvring  to 
achieve  sensor/missil*  lock-on  and  launch 
at  large  of f-boresight  angles. 

In  order  that  the  task  decomposition  is 
split  into  manageable  proportions,  each 
phase  is  divided  into  segments.  BVR  Combat 
may  constitute: 

o  Target  Detection  a  Identification; 
o  Evaluation,  Prioritisation  a  Decision; 
o  Pre-launch  Manoeuvre; 
o  Launch  Weapons; 
o  Post-launch  Manoeuvre. 

These  segments  define  the  level  at  which  a 
detailed  task  breakdown  can  be  generated. 

3.3  Task  Breakdown 

Each  distinct  phase/segment  (including  mode 
variations)  can  then  be  split  down  into  a 
list  of  discrete  tasks  that  have  to  be 
carried  out  from  start  to  finish  of  the 
phase.  The  major  problem  with  the  task 
breakdown  appearing  as  a  list  is  that  the 
impression  given  is  of  the  piloting  task 
being  very  ordered,  serial  in  nature  and 
almost  pre-ordained  in  sequence. 

Whilst  this  may  hold  true  for  strongly 
procedural  tasks  such  as  ground  procedures, 
in  Combat  for  instance,  life  is  anything 
but  straightforward! 
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Tasks  ara  re-scheduled  or  simply  ignored  as 
aission  nasds  dictate,  there  being  many 
occasions  when  the  pilot  would  wish  to  do 
aany  things  at  one  and  the  sane  tine. 

In  order  to  represent  aore  faithfully  the 
true  state  of  affairs,  the  tasks  are 
divided  into  three  categories: 

o  Primary  Tasks:  These  are  the  tasks  that 
characterise  each  segaent.  They  are 
aainly  perforaed  sequentially  and  in 
general  require  the  pilot's  foreground 
attention. 

o  Intermittent  Tasks:  These  are  performed 
by  the  pilot  as  and  when  required  or  when 
the  system  working  autonomously  requests 
pilot  intervention. 

o  continuous  Tasks:  These  are  performed 

continuously  and  concurrently,  aainly 
aonitoring  tasks,  preferably  carried  out 
by  the  systems  which  will  alert  the  pilot 
only  if  required. 

It  is  clear  that  this  gross  subdivision  of 
tasks  involves  an  initial  suggestion  of 
automation  requireaents ,  this  is  addressed 
in  the  next  section. 

3.4  Functions  Used  a  Autonation  Categories 

For  each  task,  a  function  can  be  identified 
that  is  needed  to  carry  out  the  task.  These 
functions  can  vary  froa  fully  annual  (pilot 
only)  to  fully  autoaatic  with  degrees  of 
blended  co-operation  in  between.  The 
categories  of  automation  applied  to  the 
functions  are: 

1  Manual:  purely  visual,  verbal  or  aental; 

2  Manual  Angnented:  e.g.  Fly-By-Wire; 

3  Manual  Angnented  -  Antonatically  United: 
e.g.  Anti-skid  braking; 

4  Autoaatic  -  Manually  United: 
e.g.  Autopilot  Attitude  Hold  aode; 

9  Autoaatic  -  Manual  Sanction: 

e.g.  Target  Nonination; 

6  Antonatic  Antonoaoua: 

e.g.  Systens  Status  Monitoring. 

This  requireaent  for  automation  froa  the 
operator's  point  of  view  will  feed  into  the 
detailed  systeas  design  work  which  will  be 
undertaken  using  CORE  methods  and  tools. 

3.5  Xnfornatioa  a  Control  Requireaents 

For  each  task,  the  requireaents  for 
information  presentation  and  control 
functions  within  the  cockpit  can  be 
detailed.  These  requireaents  allow  the 
pilot  to  either  perform  the  task  himself  or 
monitor  that  the  autoaatic  systeas  are 
doing  so  satisfactorily. 

This  definition  of  the  information  and 
control  requireaents  for  the  cockpit 
constitutes  the  priae  output  of  the  Mission 
Analysis.  These  elements  can  be  further 
categorised  dependent  on  the  tine  required 
to  access  the  inforaation  or  control. 

For  instance,  grouping  the  inforaation  and 
controls  into  three  groups  vis: 

o  laaediate  Access  Required, 
o  Reasonable  Access  Required, 
o  Ron-critical  Access  Required 


is  a  great  help  in  the  layout  and  aoding  of 
the  cockpit.  Such  categories  when  applied 
to  information  refer  only  to  the  data  which 
is  continuously  available  within  aircraft 
systems.  There  is  also  data  that  is 
interruptive  in  nature  and  only  arise 
following  soae  trigger  e.g.  Data  link 
messages  or  fault  warning  data.  Three 
further  categories  are  identified  to  cover 
these  cases: 

o  Immediate  Presentation, 
o  Immediate  Alert  (Reasonable  access  to 
inforaation) 

o  Alert  Required  (Non-critical  access  to 
information) . 

These  control  function  and  inforaation 
presentation  requirements  as  generated  by 
the  Mission  Analysis  are  as  yet  independent 
of  the  means  by  which  they  will  be  net  in 
the  cockpit  design.  The  above  method  is 
generic  in  the  sense  that  it  can  be  applied 
to  any  project.  The  implementation  of  these 
requireaents  is  a  very  detailed  and 
painstaking  task  which  is  project-specific 
and  must  embrace  the  areas  of  cockpit 
geometry,  layout  and  aoding. 

These  idealised  requireaents  are  also 
tempered  by  deliverable  equipment 
performance,  system  design  constraints,  an 
evolving  operational  requireaent  and 
subsequent  safety  assessaent.  The  following 
sections  report  on  the  unfolding  and 
ongoing  development  of  the  cockpit  design 
for  the  European  Fighter  Aircraft  ( EFA) . 

4  IHFORMATIOH  t  CORTROL  FPMCTIOM  ALLOCATION 

The  cockpit  requirements  have  been 
identified  within  mission  phases  upon 
selection  of  which  the  necessary 
information  and  controls  to  perform  the 
task  within  that  phase  aust  be  Bade 
available.  Where  the  Mission  Analysis  has 
indicated  a  requireaent  for  continuous 
availability  throughout  all  phases  of 
flight,  then  a  dedicated  presentation  was 
considered  in  the  cockpit  layout;  where  the 
presentation  could  be  governed  by  phase  of 
flight,  this  was  eabraced  in  the  overall 
cockpit  aoding  task  as  follows. 

4.1  Phase  of  Flight  Mo ding 

The  phases  of  flight  used  in  the  Mission 
Analysis  work  were  chosen  to  make  that  task 
aore  manageable.  In  the  actual  cockpit 
moding  work,  the  guidelines  applied  in 
determining  the  actual  mission  phases  used 
were: 

o  the  required  phase  shall  be  a  najor  and 
complete  sub-task  of  the  mission; 

o  the  number  of  mission  phases  shall  be 
kept  to  a  minimum. 

For  the  purpose  of  cockpit  moding,  the 
aission  phases  as  derived  in  the  analysis 
were  rearranged  and  reduced  in  number  thus: 


o  Ground  Procedures, 
o  Take  Off, 
o  Navigation, 
o  Coabat  (Air  to  Air), 
o  Attack  (Air  to  Surface), 
o  Approach  t  Landing. 
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These  war*  th*  phases  of  flight  ua*d  in 
allocating  th*  interaction  and  control 
functiona  to  b*  *ith*r  a  d*dicat*d 
pr*a*ntation  or  function,  or  to  b*  noded  on 
on*  of  th*  aiulti-f unction  surfaces  in  th* 
cockpit  layout. 

Noraially  a  aission  phaa*  will  b*  a*l«ct*d 
autoaatically  as  th*  result  of  a  n*c*ssary 
•v*nt ,  *.g.  Ground  Procedures  on  pow*r-up, 
Coabat  on  weapon  or  radar  air  conbat  nod* 
selection,  Attack  on  selection  of  a  weapon 
package  and  Approach  and  Landing  on 
selection  of  th*  landing  g*ar  down. 

How*v*r,  wh*r*  annual  selection  is 
required  instead  of  automatic  selection, 
th*  facility  is  provided  for  th*  pilot  to 
engage  th*  aisaion  phase  by  a  single 
action.  All  aisaion  phases  are  selectable 
froa  any  other  aisaion  phase  (except  Ground 
Procedures)  and  no  selected  phase  will  be 
changed  by  a  power  interrupt,  rurther  such 
principles  of  cockpit  noding  are  given  in 
section  5. 

4.2  Geometry  a  Layout 

There  are  aany  factors  that  govern  th* 
geoaetry  of  any  cockpit  solution;  son*  of 
which  are  directly  under  th*  influence  of 
th*  cockpit  designer,  soae  are  not.  Th* 
external  boundaries  of  th*  cockpit  in  terns 
of  canopy,  fuselage  and  pressure  bulkheads 
are  often  influenced  aor*  strongly  by 
aerodynaaic,  perforaance,  structural  and 
equipaent  installation  considerations  than 
by  cockpit  design  principles. 

The  paraaeters  which  are  defined  in  the 
cockpit  design  process  are  very  auch 
interactive  and  th*  final  choice  will  of 
necessity  be  a  coaproais*.  For  instance,  an 
upright  seat  angle  gives  an  increased  view 
over  th*  nose,  greater  depth  of  instruaent 
panel  and  a  aor*  usable  reach  envelope; 
whilst  a  aor*  reclined  seating  angle  could 
give  greater  coafort  and  "g"  tolerance. 

A  wide  percentile  range  of  pilot  sizes  has 
to  be  accommodated  which  sets  bounds  on 
reach  and  ejection  envelopes.  Th*  tools 
available  to  the  designer  in  setting  th* 
cockpit  configuration  now  include  computer 
modelling  packages  as  well  as  th*  aor* 
traditional  aannequins  on  drawing  board  and 
wooden  nock-up. 

Having  fixed  th*  cockpit  geometry,  th* 
first  level  of  th*  layout  task  is  th* 
allocation  of  crew  information  and  control 
functions  to: 

o  Manual  Data  Entry, 
o  Direct  Voice  Input, 
o  Warning  Systea, 
o  Direct  Voice  Output, 
o  Get  You  Hoa*  Instruments, 
o  Hands  On  Throttle  And  Stick  (HOTAS), 
o  Dedicated  System  Controls  a  Indicators. 

Next  cones  th*  task  of  allocating  areas 
within  the  cockpit  real  estate  to  house 
these  instruments,  indicators  and  controls. 
These  should  all  be  arranged  in  accordance 
with  th*  iaportanc*  of  th*  itea,  frequency 
of  us*  and  association  of  functions.  To  aid 
in  this  process,  th*  controls  on  th*  side 
consoles  can  be  renoted  froa  th*  controlled 
equipaent  allowing  a  aor*  ergonomic  and 
hence  optiaua  layout  of  control  panels  to 
be  realised. 


Th*  layout  is  developed  on  th*  drawing 
board  and  using  3-D  coaputer  modelling 
techniques  which  leads  to  a  full  size 
white-on-black  representation  of  th* 
cockpit  on  a  wall  board.  Th*  huaan  and 
computer  impressions  of  th*  layout  are 
amalgamated  into  a  3-D  full  size  wooden 
mock-up  of  th*  cockpit  which  is  used 
extensively  for  assessment  of  th* 
designers'  proposals  using  company  and 
customer  aircrew. 

5  COCKPIT  MODIHG 

The  aim  of  th*  cockpit  noding  is  to  provide 
the  pilot  with  th*  information  and  control 
functions  necessary  to  perform  his  tasks  in 
th*  most  effective  manner.  To  this  end  th* 
noding  is  concerned  with  how  and  when 
information  from  the  aircraft  systems  is 
presented  on  th*  multi-function  displays, 
helmet  mounted  display  or  the  get-you-hoa* 
instruments  and  how  th*  pilot  interacts 
with  these  sub-systems  using  direct  voice 
input,  th*  manual  data  entry  facility,  th* 
HOTAS  functions  or  th*  aulti-function 
controls  associated  with  the  display 
surfaces.  Failure  situations  on  th* 
aircraft  and  hazards  external  to  th* 
aircraft  both  need  to  be  alerted  to  th* 
pilot  and  .tow  these  can  be  integrated 
within  an  intelligent  warning  system  is 
also  a  product  of  th*  cockpit  aoding. 

Th*  first  step  in  this  long  and  complex 
task  is  to  establish  th*  overall  principles 
and  ground  rules  as  a  framework  within 
which  the  detailed  moding  must  fit.  These 
principles  of  cockpit  noding  cover  several 
areas  within  the  cockpit  design  process, 
the  major  elements  of  which  will  be 
considered  in  turn. 

5.1  Autonation 

Th*  automation  requirements  from  th*  user 
point  of  view  are  developed  from  the 
Mission  Analysis  work  and  operational 
experience.  Guidelines  on  how  automated 
tasks  should  be  integrated  with  th*  cockpit 
are  given  and  also  specific  areas  are 
identified  where  automation  should  be 
considered.  This  latter  consideration  has 
obvious  iaplications  for  the  system  design 
task  and  therefore  involves  much  iterative 
discussion  with  th*  system  designers. 

The  categories  of  automation  outlined  in 
th*  Mission  Analysis  have  been  simplified 
so  that  there  are  three  levels  of  control 
to  be  considered: 

o  Manually  controlled  tasks  shall  be 

initiated  by  and  involve  th*  pilot  using 
hiB  judgement  and  mental  or  physical 
capabilities; 

o  Semi-autonatically  controlled  tasks  shall 
generally  involve  co-operation  between 
th*  pilot  and  a  process.  These  tasks  nay 
require  pilot  nandat*  or  sanction; 

o  Autoaatically  controlled  tasks  can  carry 
on  without  pilot  action. 
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5.2  Display  No  ding 

Ths  large  aaount  of  information  and  control 
functions  which  hava  to  ba  managed  via  tha 
■ulti-function  display  system  cannot  ba 
displayad  sisultanaously .  For  tha  purposa 
of  intalligibility  and  for  clarity  of 
prasantation  tha  information  and  control 
functions  naad  partitioning.  This  is 
achiavad  by  allocating  information  and 
control  functions  to  mission  phases  and 
allowing  a  maximum  of  thraa  lavols  of 
prasantation: 

o  Dafault  -  Immadiata  accass  raquirad  - 
dafinas  tha  dafault  laval  of  information 
which  will  appaar  on  salaction  of  a 
mission  phasa; 

o  Laval  1  -  Reasonable  accass  allowad  - 
will  ba  availabla  from  any  laval  by  a 
maximum  of  ona  salaction  aithar  as 
information  additional  to  tha  prasant 
format  or  as  a  naw  format; 

o  Laval  2  -  Non-critical  accass  tima  - 
will  ba  availabla  from  any  laval  by  not 
mora  than  two  salactions  aithar  as 
information  additional  to  tha  prasant 
format  or  as  a  naw  format. 

Tha  split  batwaan  tha  thraa  lavals  is  a 
function  of  importanco,  priority  and 
frequency  of  usa  of  tha  information  and 
control  functions  concarnad.  Tha  limitation 
to  thraa  lavals  is  chosan  as  a  compromisa 
batwaan  tha  dasira  not  to  cluttar  tha 
display  formats  and  not  to  raquira 
axcassiva  tima  to  accass  information. 
Although  a  maximum  of  thraa  lavals  of 
information  prasantation  is  allowad,  affort 
is  baing  appliad  to  minimisa  tha  numbar  of 
implamantad  lavals  whilst  obtaining  tha 
nacassary  clarity  and  intalligibility  of 
tha  displayad  information. 

Bach  display  format  is  dafinad  as  a  sat  of 
information  alamants  tailorad  for  a 
spacific  purposa.  Information  alamants  can 
ba  addad  or  ramovad.  Tha  numbar  of  display 
formats  and  tha  hiararchy  of  accass  is 
datarminad  by: 

o  tha  datailad  allocation  of  information; 
o  tha  raquiramants  for  accass  to  lavals. 

Tha  pilot  intaracts  with  this  information 
via  tha  HOT  AS  XT  controller,  via  tha 
multi-function  controls  which  ara  arranged 
adjacent  to  tha  display  surfaces  or  by 
using  DVI .  These  control  means  serve 
various  functions  as  follows: 

o  to  select  formats; 
o  to  altar  tha  existing  information; 
o  to  add  naw  information; 
o  to  make  subsystem  mode  selections; 
o  to  show  options. 

Failura  of  ona  or  more  of  the  displays  or 
associated  controls  will  create  a 
reversionary  situation.  A  HUD  failura  will 
ba  compensated  by  flight  information 
availabla  on  tha  MHDDs.  A  MHDD  failura  will 
ba  compensated  by  reallocation  of  formats 
to  good  displays;  no  format  reconfiguration 
or  combination  is  envisaged.  Total  loss  of 
the  main  display  suite  results  in  usa  of 
tha  Oat  You  Home  displays . 


5.3  Hands  On  Throttle  And  Stick  (HOT AS ) 

Tha  HOTAS  concept  allows  improved  pilot 
performance  during  phases  of  high  workload 
or  stress  by  providing  functions  whore: 

o  Immadiata  accass  is  frequently  raquirad; 
o  Immadiata  access  is  required  during 
combat  or  high-G  manoeuvres. 

These  functions  ara  related  to  sensor 
control,  weapons  control,  flight  management 
and  defensive  aids.  Tha  operation  of  a 
HOTAS  control  provides  immediate  visual, 
aural  or  tactile  feedback  to  tha  pilot. 

5.4  Manual  Data  Entry  Facility 

This  facility  combines  the  data  entry  and 
moding  tasks  from  a  variety  of  aircraft 
subsystems  into  ona  focal  area  in  tha 
cockpit.  Its  five  main  functions  are: 

o  Subsystem  salaction  keys  dedicated  to 

-  Navigation  (waypoints  and  routes), 

-  V/UHF  Radio  1  a  2  (modes  a  freqs.), 

-  Data  Link, 

-  TACAN  (modes  and  channels), 

-  NIS  /  IFF  (modes  and  codas), 

-  Dafansiva  Aids  Subsystem, 

-  Microwave  Landing  System, 

-  Miscellaneous  (IN  a  CW  data); 
o  Moding  keys  for  task  selection; 

o  Data  Entry  Keyboard  for  alphanunarics; 
o  Two-axis  toggle  switch  for  error 
correction  and  data  sequencing; 
o  Sat  Waypoint  and  Change  Destination. 

5.5  Direct  Voice  Input 

DVI  nay  ba  used  where  appropriate  as  an 
alternative  and  parallel  moding  and  data 
entry  medium  for  interaction  with: 

o  Manual  Data  Entry  Facility; 
o  Displays  (HUD  &  MHDDs); 
o  Autopilot; 
o  Communications; 
o  Mission  phase  moding. 

Moding  or  data  entry  by  DVI  will  hava  tha 
same  impact  on  subsystem  functions  as  if  a 
manual  salaction  has  bean  made  such  that 
tha  two  interaction  methods  can  ba 
interchanged  at  any  point  in  a  routine.  Tha 
feedback  to  DVI  commands  will  always  appear 
as  for  manual  salactions  plus  haad-up 
feedback  if  implamantad.  Tha  syntax  of  tha 
vocabulary  used  by  tha  pilot  will  ba 
structured  in  order  to  balance  flexibility 
with  voice  recognition  success  rate. 

5.6  Warning  System 

Under  normal  operating  conditions  all 
onboard  aircraft  systems  will  be  subject  to 
automatic  health  monitoring  through 
built-in  test  facilities.  When  practical, 
detection,  diagnosis  and  correction  of  a 
fault  will  take  place  without  pilot 
intervention.  Each  fault  as  it  arises  will 
be  categorised  and  assigned  a  priority 
within  that  category  according  to  tha  phasa 
of  flight  and  any  other  existing  failures. 
The  aim  of  the  warning  system  is  to: 

o  Alert  the  pilot  to  a  warning  situation; 
o  inform  the  pilot  what  the  situation  is; 
o  Advise  tha  pilot  of  any  consaquancas 
and  action  that  should  be  taken. 
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To  achieve  this,  four  lavals  of  warning 
category  hava  baan  idantifiad: 

o  Catagory  1  -  Procedural  Warning; 
o  Catagory  2  -  Systaas  Warning:  Rad; 
o  Catagory  3  -  Systaas  warning:  Amber; 
o  Catagory  4  -  Procadural  Advica. 

Tha  sain  presentation  and  control  of 
warnings  inforaation  will  utilisa  visual 
attention  gattars,  attantion  gatting  sounds 
(attensons),  tha  Voica  Warning  Systaa,  tha 
Dadicatad  Warning  Panal  and  tha  MHDD 
prasantation  of  inforaation  ralating  to 
aircraw  procaduras  and  warning 
consaquancas .  A  Gat  You  Hoaa  warning  systaa 
is  also  providad  for  non-availability  of 
tha  Bain  warning  systaa. 

5.7  Gat  Tou  Hoaa  Instruaants 

Tha  raason  for  having  Got  You  Hoaa 
Instruaants  is  to  provida  tha  pilot  with  an 
indapandant  sat  of  flight  data  which  will 
anabla  hia  to  raturn  tha  aircraft  to  basa 
in  tha  avant  of  total  loss  of  tha  main 
display  suits.  Similarly,  a  nuabar  of  "high 
catagory”  warnings  will  bs  hard  wirod 
through  to  tha  Dadicatad  Warning  Panal . 

5.8  Halaat  Mounted  Sight 

Both  tha  highly  aanoouvring  anvironaant  of 
short  ranga  coabat  and  agilo  waapons  with 
largo  of f-borasight  capability  aaka  it 
assantial  that  tha  pilot  ba  abla  to  ongaga 
in  air  coabat  without  having  to  continually 
rafar  to  tha  HUD  for  waapon  aiming 
inforaation.  To  this  and,  tha  HNS  will  ba 
focusad  at  a  nominal  infinity  and  bo  usod 
aainly  for  Radar  aiaing/cuaing  of  aissilos 
and  for  autonomous  HMS  aiming  whora  tha 
aissila  is  slavod  to  tha  sight. 

5.9  Mission  Data  Loading 

All  aission-spacific  data  racording  and 
loading  will  ba  parforaad  by  aoans  of  a 
portabla  storaga  medium  so  that  manual 
input  of  aission  data  by  tha  pilot  during 
Ground  Procaduras  is  avoidad.  Typical  data 
to  ba  loadod  by  this  aoans  includes : 

o  Araaaants  package  a  configuration  data; 
o  Digital  nap  data; 
o  DVI  voica  tsaplatss; 
o  Navigation  waypoint  a  routs  data; 
o  Tactical  attack  a  defensive  data; 
o  Pilot  Sonsor  Moding  Kay  (PSMK). 

This  last  itsn,  tha  PSMK,  is  a  vary  useful 
facility  whereby  tha  pilot  can  specify 
default  values  to  certain  attributes  of  tha 
Displays  and  Controls  sub-systan  in 
accordance  with  individual  prafaranca. 

6  AIRCRKW  ASSESSMENTS 

Aircrew  hava  already  bean  involved  in  tha 
foraativa  design  process  thus  far.  Thoir 
axparionca  and  ideas  hava  bean  tapped  as 
part  of  tha  generation  of  tha  concepts, 
philosophies  and  design  proposals.  At  tha 
stage  where  tha  design  has  firaod  up 
sufficiently  to  allow  spatially 
raprasontativo  aock-ups  of  the  cockpit  and 
front  fuselage  to  ba  built,  aircraw 
assessment  of  these  facilities  bacoaas  a 
vital  interactive  and  iterative  task. 


Two  forms  of  mock-up  vehicle  are 
commissioned,  naaaly  static  and  active 
cockpits.  Use  is  also  made  of  more  generic 
prototyping  facilities.  These  three 
facilities  ara  used  for  different  purposes, 
each  is  considered  in  turn. 

6.1  Static  Cockpit 

This  cockpit  is  a  spatially  accurate 
3-dimensional  wooden  mock-up  integral  with 
a  representation  of  tha  aircraft  front 
fuselage.  It  is  fitted  with  raprasantativa 
seat,  rudder  pedals,  stick  and  throttle 
tops.  In  tha  first  instance  control  areas 
may  be  represented  by  white  on  black 
pictures  of  the  layout;  these  ara  soon 
replaced  by  the  actual  form  of  control. 

Aircrew  to  ba  used  in  the  assessments  ara 
drawn  initially  from  tha  flight  operations 
departments  of  tha  partner  companies.  Their 
anthropometric  measurements  ara  already 
knowr  and  if  possible,  a  wide  cross  section 
of  the  full  percentile  ranga  iB  used.  Later 
on  custoaer-noainated  aircraw  ara  used  to 
progress  and  approve  cockpit  developments. 
All  aircrew  are  fitted  out  in  full 
operational  clothing  as  relevant  to  the 
particular  assessment,  including  gloves, 
helmets,,  respirators,  immersion  suits, 
Nuclear,  Biological  and  Chemical  gear  and 
any  other  personal  equipment  as  required. 

With  tha  seat  adjusted  to  put  tha  pilot  at 
the  correct  design  aye  position  and  tha 
full  harness  fitted,  assessment  of  tha 
cockpit  internal  features  is  made.  Tha 
acceptability  of  tha  reach  and  vision 
envelopes  to  all  general  display  and 
control  areas  in  tha  cockpit  is  assessed  to 
agree  its  overall  configuration.  A  more 
detailed  evaluation  is  also  performed 
covering  the  necessity  for  and  location  of 
every  single  feature  in  tha  cockpit. 

Tha  pilots  are  fully  briefed  on  the 
assessment  objectives  and  procaduras, 
including  a  technical  overview  of  cockpit 
and  system  philosophies.  A  structured 
proforma  is  used  to  conduct  each  assessment 
and  to  elicit  pilot  opinion  and  rating  of 
the  particular  features  being  assessed. 

Room  is  also  allowed  to  glean  any  further 
comments  and  suggestions  which  the  pilots 
may  wish  to  make. 

At  the  end  of  each  assessment  phase,  all 
the  results  are  collated  by  the  cockpit 
group  so  that  from  the  consensus  of  pilot 
opinion,  recommendations  can  be  made  for 
changes  to  the  next  standard  of  cockpit 
layout.  These  cockpit  group  recommendations 
are  put  forward  for  agreement  at  a 
subsequent  wash-up  meeting  to  which  all 
aircrew  who  participated  in  the  assessment 
are  invited  to  attend.  In  this  iterative 
manner,  the  cockpit  layout  is  developed  and 
refined  through  agreed  standards. 

The  foregoing  has  considered  assessment  of 
the  cockpit  internal  characteristics;  in 
like  manner  external  aspects  of  the  cockpit 
are  assessed.  This  may  use  tha  same  cockpit 
shell  or  another  that  is  integral  with  the 
full-scale  aircraft  mock-up  as  appropriate. 
The  cockpit  is  mounted  at  the  correct 
height  from  the  ground  and  the  surrounding 
area  is  marked  with  a  reference  grid. 
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The  external  vision  snvslops  is  evaluated 
using  ths  graduations  on  ths  ground  and 
assessment  aada  of  any  visual  obscuration 
causad  by  axtarnal  aircraft  structura. 

This  fora  of  vahicls  is  also  fittad  vith  a 
laddar,  hand  holds  and  axtarnal  stairways 
as  appropriata  for  both  noraal  and 
aaargancy  ingrass  and  agrass  chacks  to  ba 
parforaad.  Saat  pull-out  and  ascapa 
anvalopa  chacks  ara  also  parforaad  at  this 
tiaa.  Although  not  at  prasant  tha 
rasponsibility  of  tha  cockpit  group,  thasa 
foras  of  aock-up  ara  also  usad  for 
assassaant  by  dasign  parsonnal  concarnad 
with  accass  to  aircraft  aguipaant  from  tha 
aaintainability  point  of  viaw.  with  tha 
advant  of  MANPRXNT,  this  lattar  form  of 
avaluation  is  as  important  a  part  of 
dasigning  for  huaan  intaraction  as  is  tha 
cockpit  dasign  and  avaluation  task, 

6 . 2  Active  Cockpit 

In  a  similar  way  to  that  in  which  tha 
static  cockpit  is  coaaissionad  and  usad,  at 
a  slightly  latar  stags  in  tha  cockpit 
dasign  an  activa  cockpit  is  brought  on 
lina.  Tha  intarnal  diaansions  of  tha 
cockpit  and  tha  aguipaant  with  which  it  is 
furnishad  ara  as  raprasantativa  as  thay  can 
bs  in  fora,  fit  and  function  givan  that 
auch  of  it  is  aithar  off  tha  shalf 
aguipaant  or  manufacturad  in-housa.  Tha 
cockpit  is  linkad  to  an  assassaant  control 
station  and  tha  coaputar  facilitias  that 
housa  tha  simulation  software. 

Tha  basic  facility  includes  a  linearised, 
six  degree  of  fraadoa  aerodynamic  responsa 
aodal ,  which  whan  interfaced  with  tha 
outside  world  systea  and  tha  incaptors  in 
tha  cockpit,  enables  tha  pilot  to  fly  tha 
simulation  and  receive  realistic  visual 
cues.  Provision  of  aircraft  system  models 
and  interactive  displays  and  controls 
allows  pilot  assassaant  of  the  cockpit  via 
aission-capabla  simulation. 

Aircrew  usad  for  assessments  in  tha  activa 
cockpit  ara  drawn  fron  tha  saaa  pool  as 
parforaad  tha  static  cockpit  assessments. 
Vary  siailar  procedures  ara  usad  for 
briefing,  conducting  and  reviewing 
evaluations  in  the  activa  cockpit.  It  is 
vital  that  tha  aircrew  provided  for  this 
iterative  avaluation  and  development 
process  ara  fully  raprasantativa  of  tha 
final  and  user  of  tha  aircraft  in  terms  of 
relevant  physical  and  mental  attributes. 

This  fora  of  active  cockpit  is  now  seen 
very  auch  as  a  dasign  tool  in  its  own 
right.  It  is  coaaissionad  and  usad  as  early 
on  in  the  project  as  possible  providing 
auch  useful  information  on  parameters  and 
aoding  to  ba  incorporated  in  tha  developing 
design.  It  also  gives  increased  confidence 
on  tha  acceptability  of  proposed  concepts 
and  thus  provides  a  risk-reduction 
function,  so  important  to  tha  system 
developer  in  a  fixed  price  contract.  This 
vehicle  is  tha  priaa  aaans  by  which 
acceptable  usar-in-tha-loop  performance  is 
demonstrated;  this  Lay  ba  a  contractual 
obligation  in  the  ara  of  MANPRXNT. 


6.3  Rapid  Prototyping 

As  wall  as  facilitias  that  ara  specifically 
geared  to  one  project,  there  ara  more 
generic  tools  that  have  wider  application. 
One  such  tool  is  the  Generic  Rapid 
Prototyping  Tool  that  has  bean  developed 
in-house  from  commercially  available 
eguipment .  Tha  GRPT  has  been  usad  on  EPA  in 
parallel  with  tha  whole-cockpit  assessments 
to  investigate  certain  aspects  of  tha 
interface  off-line;  solutions  that  have 
particular  merit  ara  fad  back  into  tha 
activa  cockpit  for  final  avaluation. 

This  approach  has  bean  usad  to  prototype 
novel  formats;  to  rework  existing  formats 
that  have  bean  found  unacceptable;  to 
investigate  tha  interaction  of  tactical 
symbology  with  colour  nap  background;  and 
to  optimise  tha  control  laws  for  a  display 
cursor  driven  under  KOTAS  XT  control.  The 
tool  allows  rapid  iterations  of  concepts  as 
whole  formats  can  be  developed  in  days, 
changes  handled  in  hours  and  thus  coapleta 
assessments  carried  out  in  a  few  weeks. 

The  GRPT  has  bean  used  to  such  good  affect 
that  it  is  being  considered  as  an  ideal 
means  of  capturing  tha  reguirement  for  a 
particular  MMI  at  tha  earliest  stages  of  a 
project.  Customer  involvement  in  this 
activity  is  essential  so  that  early 
agreement  on  tha  fora  and  faal  of  tha 
interface  can  ba  obtained. 

7  SAFETY  ASSESSMENT 

Whilst  on  the  one  hand  tha  dasign  of  a 
fighter  aircraft  is  optimised  for  war-tine 
performance,  on  tha  other  hand  tha  issue  of 
system  safety  is  also  a  priae  consideration 
especially  for  peace-time  flying.  There  are 
complex  tradeoffs  to  be  performed  in 
achieving  acceptable  levels  of  both 
parameters  recognising  that  enhancement  of 
one  may  compromise  the  other. 

A  relatively  high  proportion  (40%)  of 
catastrophic  fast  jet  losses  are  attributed 
to  pilot  error.  This  appears  to  be  a 
convenient  catch-all  for  accidents  caused 
by  inadeguate  training  or  ill-defined 
operating  procedures  or  bad  design  of  the 
cockpit  interface  which  exacerbated  a 
problem  in  a  high-stress  situation.  The 
insidious  nature  of  system-induced  pilot 
error  is  also  worthy  of  close  examination. 

The  issue  of  safety  within  the  cockpit  is 
therefore  much  more  than  mere  consideration 
of  the  physical  aspects  of  the  MMI;  well 
established  procedures  exist  for  performing 
hazard  analyses  on  hardware  and  software 
functions.  It  is  also  more  than  reviewing 
the  likelihood  of  aircrew  error  although 
this  is  difficult  enough  in  itself.  There 
is  an  overlap  between  the  two  areas  where 
the  Interaction  of  the  Man  with  the  Machine 
is  more  important  than  the  Interface. 

It  is  unrealistic  to  look  for  this 
interaction  being  error-free;  but  it  is 
important  that  the  reguired  interaction  is 
as  error-tolerant  as  possible. 
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Safety  assessment  of  this  nature  is  in  its 
infancy  with  no  widely  availabla  methods  or 
procaduras  for  carrying  it  out.  A  method  of 
hasard  analysis  and  safaty  assassaant  for 
tha  cockpit  has  baan  darivad  for  usa  on  EFA 
which  addrassas  tha  issuas  raisad  abova. 

In  general  tha  approach  has  baan  to  taka 
aach  information  clamant  and  control 
function  within  tha  cockpit  and  consider 
tha  hasard  that  could  occur  if  tha  pilot: 

o  incorrectly  selects  tha  control; 
o  inadvertantly  selects  tha  control; 
o  fails  to  select  tha  control  as  required; 

o  misreads  a  displayed  item; 
o  mistakes  one  item  for  another; 
o  fails  to  saa  a  displayed  item. 

For  aach  of  those  avantualitios  a  detailed 
knowledge  of  its  impact  on  aircraft  system 
operations  is  required  to  assign  a  severity 
to  tha  prospective  worst-case  scenario.  Tha 
way  in  which  tha  function  has  baan 
implomantad  in  tha  cockpit  design  is  next 
considered  to  give  a  qualitative 
probability  that  tha  error  might  occur.  Tha 
hasard  severity  and  risk  assessment 
(frequency  of  occurrence)  are  than  used  to 
assign  a  safaty  risk  laval  to  the  function. 

Each  function  than  has  an  assigned  safaty 
risk  level  of  High  (unacceptable).  Moderate 
(acceptable,  subject  to  review)  or  Low 
(acceptable  without  review)  as  appropriate. 
A  detailed  justification  is  given  for  tha 
assignment  of  each  safety  risk  level.  If 
the  safety  risk  is  unacceptable  then  action 
must  be  taken  to  reduce  the  risk.  This 
action  will  depend  upon  the  nature  of  the 
error/failure  but  should  consider: 

o  Redesign; 

o  Incorporate  safety  devices; 
o  Incorporate  warning  devices; 
o  Apply  specific  procedures  or  training. 

The  results  of  this  cockpit  safety 
assessment  along  with  similar  assessments 
of  all  the  other  systems  on  the  aircraft 
feed  into  an  overall  aircraft  system  safety 
assessment  where  more  global  safety  issues 
are  addressed.  The  development  and 
application  of  this  semiformal  method  for 
safety  assessment  of  the  cockpit  is  very 
much  "hot  off  the  presses"  and  therefore 
still  subject  to  review  and  improvement. 

•  comcLosioms 

EFA  will  benefit  greatly  as  a  potent  Weapon 
System  from  the  structured  approach  taken 
to  both  cockpit  and  system  design.  By 
virtue  of  this  approach  and  the  harnessing 
of  appropriate  Human  Factors  expertise, 
methods  and  tools,  the  EFA  cockpit  will  be 
a  flexible  workplace  that  allows  efficient, 
reliable  and  safe  human  operation  with  a 
manageable  pilot  workload.  In  the  light  of 
current  EFA  experience,  the  following 
conclusions  can  also  be  drawn: 


Optimisation  of  the  weapon  system  design 
can  only  be  realised  if  a  common  approach 
is  taken  to  the  interpretation  and 
implementation  of  the  customer  requirement 
in  all  design  areas.  This  is  nowhere  more 
important  than  in  the  integration  of  design 
approach  to  system  and  cockpit  functions. 


A  structured  approach  to  the  design  of  an 
integrated  vehicle  that  considers  the 
hardware,  software  and  human  together  as  a 
system  is  vital  in  realising  enhanced 
weapon  system  performance  whilst  managing 
the  pilot  workload  problem. 


Structured  system  design  methods  and 
mission  analysis  /  task  analysis  methods 
must  be  knit  together  as  part  of  an 
integrated  toolset.  BAe  is  committed  to  the 
development  of  an  integrated  design  process 
that  allows  all  the  attendant  benefits  to 
be  realised. 


Typical  cockpit  design  methods  and  tools 
have  equal  application  to  the  design  of 
every  part  of  the  aircraft  with  which  there 
is  human  interaction.  The  application  of 
cockpit-oriented  Human  Factors  skills  and 
experience  to  other  aspects  of  system 
design  is  essential  in  fulfilling  the 
requirements  of  initiatives  such  as 
MANPRINT. 
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Discussion 


QUESTION  G.H.  HUNT 

How  many  "Forcing  Missions"  do  you  need  to  analyse  to  ensure  that  the 
resultant  cockpit  design  is  not  specifically  optimised  to  a  limited  range 
of  operational  roles? 

REPLY 

You  need  to  limit  the  full  analysis  to  only  a  few  (3  or  4)  "Forcing 
Missions"  to  contain  the  task  to  a  reasonable  level.  It  is  then  important 
to  validate  the  resultant  design  by  testing  that  design  against  other 
missions  to  satisfy  the  designer  that  the  chosen  "Forcing  Missions"  were, 
indeed,  the  most  demanding  tasks. 

QUESTION  M.  JACOBSEN 

Was  the  task  of  Eurofighter  cockpit  design  shared  by  the  4  nations  or 
was  it  the  sole  responsibility  of  B  Ae? 

REPLY 

The  4  nations  are  involved  in  the  cockpit  design  process  and  are 
represented  on  the  cockpit  assessment  working  group. 
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CVA,  Cockpit 

Design  and  Development  Tool 
by 

Dipl. -Math.  Christoph  Weber 
Elektronik-System-Gesellschaft  mbH 
Vogelweideplat2  9 
8000  Munchen  80 
Germany 


Suwary 

The  conceptual  design  and  development  of  a 
modern  helicopter  cockpit  requires  the 
consideration  of  ergonomic,  operational  and 
technical  aspects.  New  and  additional  tech¬ 
nologies  lead  to  a  steady  increase  of  data 
and  workload,  so  that  an  essential  task  is 
to  obtain  an  optimum  layout  of  the  "Man- 
Machine-Interface"  (MMI).  For  the  perform¬ 
ance  of  this  task  MoD  has  charged  ESG  to 
generate  a  national  Cockpit  Design  Tool 
CVA  and  to  operate  it  in  parallel  with  the 
TIGER  development.  Due  to  the  national 
character  of  the  CVA  those  tasks  are  pri¬ 
marily  handled  which  concern  the  specific 
German  portion  of  the  TIGER  program,  e.g. 
the  control  and  display  system  of  the 
digital  map  generator  the  helmet-mounted 
sight/display  system  the  HF  data  link. 

These  complex  systems,  however,  cannot  be 
investigated  separately  due  to  the  multiple 
reciprocal  actions  with  the  remaining  dis¬ 
play  and  control  system,  but  have  to  be 
considered  with  the  overall  cockpit. 

Thus  the  CVA  is  the  reproduction  of  a  func¬ 
tioning  1:1  cockpit  of  the  PAH 2  version  of 
the  Tiger  and  is  operated  in  close  coopera¬ 
tion  with  the  future  user  (pilot).  It  is  a 
closed-loop  simulator  which  enables  the 
checking  of  important  areas  of  the  MMI  not 
only  in  theory,  but  mainly  under  practical 
conditions  long  before  a  prototype  of  the 
new  helicopter  exists. 

1 .  Problem  Situation 

The  conceptual  design  and  development  of 
new  helicopters  and  thus  of  new  modern  cock¬ 
pits  always  requires  a  symbiosis  between 
most  recent  technology  and  man  in  the  cock¬ 
pit.  In  this  connection  the  emphasis  was 
placed  on  the  technology  and  not  on  the 
crew  which  is  directly  involved  with  the 
operation  of  the  system,  by  tasks  like 
helicopter  command  and  control  and  mission 
performance.  Moreover  the  physical  charac¬ 
teristics  and  the  intellect  of  man  have 
hardly  changed  as  opposed  to  the  techno¬ 
logy  so  that  man  tends  to  become  the 
weakest  link  in  the  "Man  Machine  System". 

The  primary  objective  must  be  today  to 
relieve  the  crew  in  the  cockpit.  Thus  the 
crew-system  interface  is  an  essential  key 
for  not  exceeding  the  limits  of  psychical 
and  physical  crew  stress  also  under  extreme 
conditions  such  as 

-  Lowest  flight 

-  Night  missions 

-  Operation  under  extremely  bad  visibility 
conditions 


The  problem  becomes  even  more  obvious  by 
fact  that  an  enormous  increase  in  data 
volumes  as  the  result  of  new  additional 
sensors  and  inclusion  in  command  and  con¬ 
trol  systems  leads  to  a  steady  data 
quantity  and  workload  increase,  thus 
exceeding  the  limits  of  error  free  data 
processing  by  man. 

The  design  and  layout  of  a  future  cockpit 
like  for  the  TIGER  is  influenced  by  ergo¬ 
nomic,  operational  and  technical  aspects, 
whereby  an  essential  task  is  the  optimum 
layout  of  the  "Man-Machine-Interface" 
(MMI),  i.e.  to  present  to  the  crew  in  the 
cockpit, 

at  the  right  time, 

in  the  right  presentation, 

the  right  information, 

and  to  design  the  control  structure, 

uncomplicated, 

uniform, 

error  tolerant. 

An  essential  aspect  in  this  connection 
which  is  to  be  considered  is  that  the  per¬ 
fection  of  the  control  and  display  systems 
is  not  provided  if  nothing  can  be  added, 
but  if  nothing  can  be  taken  away  any  more. 

In  order  to  be  able  to  perform  this  task 

MMI  Optimization 

for  the  future  TIGER  version  PAH 2  effec¬ 
tively  and  cost-efficient  the  experimental 
program 


Figure  1:  CVA  a  national  Cockpit  Design 
and  Development  Tool 
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CVA  Cockpit-Versuchs-Auf '  au 

is  performed  in  parallel  with  the  develop¬ 
ment. 

2.  Tasks  and  Objectives 

The  CVA  purpose  is  as  follows: 

-  gaining  experimental  information  on  the 
man-machine-interface  performance 
capability 

-  optimizing  the  development  result,  the 
development  sequence  with  regard  to  time 
and  cost  aspects 

-  limiting  the  development  risk 


experience  of  the  future  users,  the  test 
pilots  and  flying  instructors  in  due  time. 

Thus  the  CVA  is  an  effective  tool  with 
which  future  users  and  system  engineers  can 
check,  long  before  a  prototype  of  a  new 
helicopter  exists,  important  areas  on  the 
MMI  not  only  m  theory,  but  mainly  under 
practical  conditions. 

The  strict  realization  of  the  conceptual 
guidelines 

-  modularity, 

-  rapid  reconfiguration  possibility, 

-  rapid  modification  capability. 


Figure  2:  Main  Tasks  of  the  CVA 

The  essential  task  of  the  CVA  design  and 
development  tool  is  the  conversion  of 
theoretical  concepts  to  experimental  hard¬ 
ware  for  investigating  the  MMI  for  the 
future  TIGER  under  quasi-real  operational 
conditions. 

Thus  the  CVA  comprises  the  following  tasks: 

-  Specification  of  functions,  control 
sequences,  displays,  arrangements,  pro¬ 
cedures  etc.  for  the  PAH2  cockpit  due  to 
information  gained  experimentally 

-  Checking  of  specified  specifications  and 
developments  of  the  PAH2  cockpit 

-  Verification  of  theoretical  proposals  for 
control  and  display  functions 

o  for  crew  communication  in  a  tandem 
cockpit 

o  for  workload  relief  in  a  tandem  cockpit 

In  addition  to  ergonomic  questions  and  re¬ 
quirements  of  a  cockpit  crew-supporting 
functions  are  tested  experimentally  taking 
into  consideration  new  display  and  control 
technologies . 

In  order  to  enable  a  CVA  to  fullfil  these 
objectives  and  tasks  the  experimental  set¬ 
up  must  be  flexible  and  modular  with  respect 
to  hard-  and  software,  in  order  to  allow 
the  rapid  and  economic  investigation  of 
alternative  configurations  resp.  functions. 

3 .  Approach 


have  guaranteed  that  the  CVA  is  ready  for 
11-12  experimental  campaigns  per  year. 

The  CVA,  of  course,  cannot  and  is  not 
intended  for  replacing  flight  trials  com¬ 
pletely.  It  remains  a  fact  that  the  degree 
of  confidence  in  the  results  obtained  by 
flight  trials  is  higher  than  that  obtained 
by  results  derived  from  a  CVA,  although 
flight  trials  are  very  time-consuming  and 
expensive.  With  regard  to  the  latter 
aspects,  the  obvious  advantage  is  offered 
by  a  support  tool  like  the  CVA,  as  it 
enables  the  performance  of  the  tests 

-  without  flying  authority 

-  irrespective  of  the  weather  conditions 

-  with  non-flight-qualified  cheap  com¬ 
mercial  equipment 

-  without  consideration  of  environmental 
critical  and  flight  safety  critical 
aspects 

For  these  reasons  the  CVA,  as  a  link  between 
the  developing  source  and  the  user,  is  an 
important  effective  and  cost-favourable 
design  and  development  tool  enabling  the 
limitation  of  the  risks  associated  with  a 
complex  cockpit  development. 

4.  Description  of  the  CVA 

The  CVA  is  a  stationary  duplicated  full- 
scale  fully  functioning  PAH2  cockpit,  i.e. 
a  tandem  arrangement  of  the  crew  work  sta¬ 
tions  with  the  respective  control  f.nd  dis¬ 
play  facilities,  such  as: 

-  Multi  Function  Displays  (MFD) 

-  Helmet  Mounted  Sight/Display  System 
( HMS/D ) 

-  Control  and  Display  Units  (CDU) 

-  Radios  Frequency  Indicator  (RFI) 

-  Weapon  control  elements 

-  Backup  instruments 

In  addition  to  the  actual  cockpit  setup  the 
design  and  development  tool  CVA  includes: 

-  A  simulation  facility  for  the  not  yet 
available  original  digital  map  unit 

-  A  low-cost  external  vision  unit 

-  A  sensor  vision  facility 

-  A  target  generation  facility 

-  A  test  engineer  stand 


In  order  not  to  face  the  crew  with  additi¬ 
onal  demands  as  a  result  of  the  new  tech¬ 
nologies,  but  to  enable  the  latter  to  place 
its  demands  on  the  future  technologies 
involved,  ESG  practises  a  cockpit  develop¬ 
ment  approach  which  also  considers  the 


For  the  control  of  the  system  efficient 
simulation  computers,  interface  computers 
and  symbol  genera^rs  are  used. 

In  the  following  the  most  important  control 
and  display  elements  are  briefly  described. 
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The  primary  control  facilities  cyclic 
stick,  collective  stick  and  pedals  can  be 
adjusted,  like  the  seats,  individually  to 
the  respective  trial  crew.  Furthermore  the 
control  facilities  between  the  front  and 
rear  cockpit  can  be  influenced  by  a  test 
engineer.  Thus  it  is  possible  to  investi¬ 
gate  the  crucial  subject  crew  communica¬ 
tion  in  a  tandem  cockpit. 

The  MFD's  include  flight  control,  naviga¬ 
tion,  position,  weapon  and  system  informa¬ 
tion,  whereby  this  information  can  be  dis¬ 
played  as  a  function  of  the  mission, 

-  continuously, 

-  as  required, 

-  automatically. 

Due  to  the  special  mission  conditions  of 
the  TIGER  the  simulation  of  the  mission 
phase  target  acquisition,  target  engagement 
plays  an  essential  part.  In  this  phase  the 
sensor  information  is  displayed  to  the  crew 
overlayed  with  synthetic  symbology  for 
flight  control  and  weapon  employment  on  the 
HMD . 

The  central  Control  and  Display  Unit  (CDU) 
is  located  in  the  left  front  console  area 
(front  and  rear  cockpit). 

It  is  the  main  input  unit  and  is  primarily 
used  for  controlling  the  following: 

Flight  Management  Functions, 

Communication, 

Navigation, 

Digital  input  of  system  data. 

With  the  help  of  the  Line  Selector  and  Fix 
Function  Keys  information  on  the  MFD's  is 
added/cancelled  as  a  function  of  the  situa¬ 
tion. 


4.1  Facility  for  the  Simulation  of  the 
Digital  Map  Unit 


Figure  3:  Simulation  Facility  for  the 
digital  Map 

This  facility  enables  the  simulation  of  the 
functions  of  the  future  digital  map  unit 
which  is  required  by  the  PAH2  on  the  dis¬ 
plays  for  aerial  navigation  and  display  of 
the  tactical  situation.  Tne  controlling  of 
the  map  takes  place  as  a  function  of  the 
location  and  heading  of  the  helicopter. 

The  map  scales  1  :  50  000,  1  :  100  000, 

1  :  250  000,  1  :  500  000  and  a  zoom  func¬ 
tion  are  realized.  On  the  actual  map 
representation  the  information  on  the 
tactical  situation,  flightpath  control, 


obstacles  are  superimposed.  This  informa¬ 
tion  is  summarized  in  so  called  thematic 
classes . 

In  addition  to  the  actual  map  function  the 
demanded  Video  Memory  Function 


Simulation  Video  Memory 

Processing 


Figure  4:  Video  Memory  Function 


is  simulated.  It  permits  the  evaluation  of 
stored  sensor  video  individual  images  resp 
sequences  (in  cooperation  with  the  sensor 
vision-target  generation  facility)  and  the 
take  over  of  targets  in  the  map  represen¬ 
tation  . 


4.2  Low-Cost  External  Vision 


On  the  basis  of  digital  terrain  data  a 
simple  computer-generated  external  vision 
is  generated  which  is  adapted  to  the  CVA 
objective.  The  term  "simple"  here  refers 
to  the  degree  of  detail  of  the  terrain 
representation.  The  low-cost  external 
vision  is  used  for  flying  from  place  A  to 
B. 


4.3  Sensor  Vision  Facility 

The  sensor  vision  facility  consists  of  a 
landscape  generator  which  generates  on  the 
basis  of  digitized  landscape  images 
reality-true  synthetic  computer  pictures, 
as  if  a  pilot  or  commander  would  see  the 
landscape  through  his  sensor.  The  degree 
of  detail  of  the  sensor  vision  is  of  very 
high  photographic  quality  as  opposed  to 
the  low-cost  external  vision.  The  field  of 
view  (FOV)  available  is  as  follows: 

30  0  x  30,0 
5,0  x  5,0 
2,5  x  2,5 


'Che  pilot  and  commander  can  select  between 
the  FOV 1 s  indicated  independently  from  each 
other,  depending  on  the  task. 

Furthermore  the  representation  as  TV  or  TI 
image  in  different  sensor  picture  qualities 
is  possible,  whereby  the  sensor  picture 
quality  can  be  varied  online  by  the  test 
engineer  in  charge.  In  accordance  with  the 
sensor  arrangement  (Mast, Bug  mounted)  of 
the  TIGER  two  evepoints  are  realized  at  an 
altitude  difference  of  approximately 
1  mtrs.  The  viewing  directions  of  the  pilot 
and  commander  are  independent  from  each 
other,  and  are  controlled  by  the  respective 
line  of  sight  of  the  HMS  system,  resp.  by 
a  ]oyst ick/Gunner  Armament  Grip  (GAG). 

The  sensor  vision  facility  is  used  in  the 
course  of  target  acquisition,  identifica¬ 
tion,  fighting,  i.e.  if  the  helicopter  is 
hovering  and  the  crew  requires  a  detail- 
true  external  view. 

4.4  Target  Generation  Facility 

The  target  generator  is  used  for  the  visual 
representation  of  targets  so  that  -  in  com¬ 
bination  with  the  landscape  generator  -  the 
crew  receives  a  realistic  representation  of 
the  position  area  including  ground  and  air 
targets.  The  system  is  designed  such  that 
up  to  four  targets  can  be  activated  simul¬ 
taneously  and  can  be  controlled  by  a  test 
engineer.  Thus  a  high  flexibility  is 
reached,  in  order  to  generate  for  the  crew 
a  practice-oriented  workload.  A  suitable 
video  switching  faciLity  enables  the  crew 
to  switch  the  sensor  image  of  the  respec¬ 
tive  other  crew  remember  to  the  own  MFD. 

4.5  Test  Engineer  Stand 

The  test  engineer's  stand  is  designed  such 
that  the  following  functions  can  be  per¬ 
formed  : 

-  Simulation  of  a  ground  radio  station 

-  Control  of  the  test  sequences 

-  Supervision  of  the  test  sequences 

-  Stimulation  of  system  functions 

For  that  'he  test  engineer  has  three  MFD’s, 
one  RF1  and  a  data  input  unit.  Thus  he  is 
in  a  position  to  indicate  for  him  one  of 
the  information  presently  shown  in  the 
cockpit  MFD 1 ,  MFD2  and  HMD  pllot/MFD4  and 
HMD  commander)  including  the  respectively 
current  radio  frequencies. 

Furthermore  'he  computer  configuration,  its 
performance  capability  and  the  modular 
f  met  tonal  set  ip  offer  adequate  flexibility 
and  growth  po'en'ial  for  modifications  and 
ex'ens  ions  . 

5.  CV A  Functions 

For  - ne  dynamic  ;losed  loop  operation  of 
'he  "/A  all  necessary  functions 


Simulation  Functions 


Figure  5:  Overview  of  Simulation  Function 


were  realized,  whereby,  due  to  the  large 
number  and  complexity  of  these  functions 
only  a  rough  survey  is  given  here. 

The  CVA  functions  include: 

-  Display  functions 

Display  and  symbol  generator  functions 

Sensor  vision 

External  vision 

Optical  warnings 

Acoustic  warnings 

-  Control  functions 
Helicopter  control 
CDU-f unctions 
Weapon  control 
Visionics 

Map  unit  control 

-  Weapon  system-specific  simulation  func¬ 
tions,  such  as 

Helicopter  dynamics 
Engine  functions 
Navigation 
Flight  management 
Mode  management 
Weapon  functions 
Map  unit  management 
Warning  functions 
Data  link 

-  Basic  simulation  functions,  such  as 
Test  and  control  functions 
Stimuli  functions 
Recording/evaluation  functions 

The  necessary  simulation  functions  are  sti¬ 
mulated  by  the  inputs  of  the  crew  (test 
engineer)  and  the  respective  information  is 
shown  on  the  display  units  in  the  cockpit, 
so  that  the  test  crew  is  part  of  the  CVA 
closed  loop  simulation. 

In  the  implementation  of  the  functions 
attention  was  paid  to  modularity  so  that  it 
is  guaranteed  to  mod i f y/reconf igure  the 
system  within  short  times. 

6.  Test  Implementation 

Since  autumn  1990  the  CVA  has  been  used  by 
trie  national  users.  In  1991  eleven  one- 
week  test  campaigns  on  different  subjects 
'ook  place.  For  1992  another  eleven  test 
campaigns  are  intended.  In  the  further  (at 
present  booked  out  until  1995)  future  use 
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of  the  CVA  program-oriented  extensions  and 
modifications  are  planned.  In  the  following 
a  short  survey  on  the  subjects  handled  so 
far  in  the  CVA  is  provided. 

The  main  areas  of  test  are  as  follows: 

1.  Control  structure  and  control  functions 
of  the  digital  map  unit 

2.  Comparison  of  two  different  CDU  control 
surfaces 

3.  Special  symbologies  for  the  HMD 

To  1: 

This  complex  work  package  concerns  the  con¬ 
trol  and  display  elements  such  as:  MFD's, 
CDU,  GAG,  visionic  panel,  sticks.  As  essen¬ 
tial  tasks  of  the  commander  the  following 
was/is  dealt  with: 

Control  sequences 

Application  of  the  thematic  classes  of  the 
digital  map  unit 

EDIT  functions  of  the  digital  map  unit 
ZOOM  functions  of  the  digital  map  unit 
Data  link 

Error  messages  of  the  system 
Video  memory  management  functions 

In  the  course  of  the  experiments  with  the 
pilots  the  weak  points  of  the  theoretical 
specifications  were  generated  and  as  a 
result  an  optimized  control  and  display 
alternative  was  presented,  which  meets  the 
operational  requirements. 

To  2: 

The  subject  of  discussion  was  a  CDU  control 
surface  with  a  single  (smaller)  and  a 
double  function  keys  (larger)  keyboard. 

The  experimental  investigations  under  par¬ 
ticipation  of  eleven  pilots  in  a  quasi- 
real  environment  were  to  prove  with  which 
one  it  was  possible  to  enter  faster  and 
more  error-free  formatted  text  such  as  way- 
points,  frequencies  and  unformatted  text 
such  as  free  texts.  The  experiments  took 
place  with  and  without  ABC-protective 
gloves  and  with  and  without  additional  work¬ 
load  of  the  operator.  The  evaluation  clear¬ 
ly  showed  that  the  single  keyboard  despite 
smaller  keys  can  be  better  operated  also 
with  ABC  gloves. 


To  3: 

The  experiments  on  HMD  symbology  are  pre¬ 
sently  performed  in  the  CVA  with  a  monocu¬ 
lar  HMD  (FOV  30  x  40).  The  experiments  are 
broken  down  acc.  to: 

-  HMD-symbology  for  flight  guidance 

-  HMD-symbology  for  weapon  employment 

At  present  the  symbology  for  flight  guid¬ 
ance  is  investigated.  First  results  after 
two  experimental  campaigns  have  clearly 
shown  that  the  theoretical  specifications 
shows  a  strong  overloading  of  the  HMD  sym¬ 
bology.  Therefore  alternatives  were  de¬ 
veloped  and  tested  with  the  pilots,  for  the 
purpose  of  a  reduction  of  the  most  import¬ 
ant  aspects.  Furthermore,  like  in  the  case 
of  the  MFD  additional  information  can  be 
added/cancelled  as  a  function  of  the  situ¬ 
ation,  whereby  the  HOCAS  has  to  be  con¬ 
sidered  here.  Final  results  will  be  avail¬ 
able  in  the  course  of  1992.  It  is  intended 
to  continue  investigations  with  a  binocular 
helmet  system  still  in  1992. 

The  evaluation  on  1  -  3  was  performed: 

-  subjectively  by  questioning  (Cooper/ 
Harper-Rating)  and  collection 

-  objectively  by  the  determination  of 
control  times  and  error  frequencies. 

Furthermore,  if  required,  experts  of  the 
Flight  Medical  Institute  in  Manching  were 
included  in  the  investigations. 

7.  Conclusion 

The  design  and  development  tool  CVA  per¬ 
mits,  in  close  cooperation  with  the  user, 
to  specify  experimentally  the  requirements 
of  a  modern  tandem  cockpit,  to  convert 
theoretical  specifications  and  to  verify 
them,  and  to  guarantee  the  optimization  of 
the  control  and  display  functions  for  work 
reduction,  so  that  the  following  is 
guaranteed  for  the  cockpit  development: 

-  Adaptation  of  the  technology  to  man 

-  Reduction  of  the  control  steps 

-  Integrated  situation-adapted  information 
representation 

-  Automatisation  of  routine  tasks 


CDU  Comparison 
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Figure  6:  CDU- Keyboard  Comparison 
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1.  SUMMARY 

Growing  numbers  of  targeting  information  sources  in 
military  cockpits  have  led  to  the  development  of 
automatic  target  recognition  (ATR)  systems.  Since 
initially  most  ATRs  will  be  used  with  an  operator,  a 
series  of  experiments  was  conducted  to  investigate 
aspects  of  the  interface  between  the  operator  and  the 
ATR.  Two  experiments  are  described  which  measured 
the  speed  and  accuracy  of  ship  identifications  made  by 
an  operator  using  information  from  an  imaging  sensor 
and  a  simulated  ATR.  These  measures  were  compared  to 
performance  of  the  unaided  operator  and  to 
performance  of  the  autonomous  ATR.  The  accuracy  of 
the  ATR,  the  format  of  its  output,  and  the  quality  and 
type  of  the  sensor  information  was  varied  in  the 
experiments.  The  results  are  discussed  in  terms  of  their 
implications  for  the  design  of  the  operator-ATR 
interface  that  will  lead  to  satisfactory  system 
performance. 


Workload  problems  are  especially  great  during  the 
targeting  phase  of  the  mission  which  extends  from 
target  detection  to  weapon  release.  During  targeting, 
when  the  aircrew  really  needs  to  have  all  of  the 
information  available,  a  major  concern  is  whether  the 
crew  members  will  have  sufficient  attentional  and 
mental  reserves  to  use  information  from  these 
additional  sources  effectively.  Thus  the  dilemma 
discussed  in  1982  remains.  On  the  one  side, 
information  is  provided  to  allow  the  expansion  of 
mission  capabilities.  On  the  other  side,  the  operator  is 
already  too  busy  to  use  it  optimally.  As  was  the  case 
ten  years  ago,  new,  more  sophisticated  automated 
systems  are  proposed  as  solutions.  Some  of  these 
systems  will  take  over  the  aircrew's  routine  operating 
and  monitoring  tasks.  Others  will  assist  the  pilot  in 
processing  and  integrating  information  and  sensor 
data  from  multiple  sources.  Still  others  will  vary  the 
degree  of  automation  to  accommodate  variations  in 
workload. 


2.  INTRODUCTION 

Ten  years  ago  the  Advisory  Group  for  Aerospace 
Research  and  Development  (AGARD)  published  the 
Proceedings  of  a  conference  on  the  Human  Factors . 
Considerations  in  High  Performance  Aircraft  (Ref.  1). 
The  1982  conferees,  noting  the  exponential  growth  in 
the  number  of  cockpit  displays  and  controls  in  combat 
aircraft,  were  seeking  ways  to  control  pilot  workload. 
They  saw  multifunction  controls  and  displays,  new 
display  technologies,  increases  in  the  capacity  and 
speed  of  onboard  computers,  and  the  increasing  use  of 
automation  as  the  principle  means  of  benefiting  from 
the  available  equipment  and  information.  Many 
concluded  that  even  more  information  and  equipment 
could  be  induded  in  future  avionics  suites  if  their 
designs  were  based  on  good  human  factors  principles 
and  good  models  of  human  cognitive  abilities. 


2.1  Background 

Many  attempts  are  underway  to  develop  automated 
systems  that  process  and  fuse  targeting  information. 
These  systems  are  generally  known  as  Automatic 
Target  Recognizers  (ATRs),  whether  they  are  designed 
to  identify,  to  detect,  to  track  a  target,  or  to  do  all 
three.  In  fact,  the  design  goals  of  different  ATRs  vary 
tremendously.  Some  are  meant  to  carry  out  the  entire 
range  of  targeting  tasks  from  detection  to  weapon 
release  without  operator  involvement.  Others  are  more 
limited  in  scope,  designed  to  do  only  one  of  the 
targeting  functions,  e.g.,  detection  or  identification. 
ATRs  also  vary  in  the  nature  and  amount  of  the 
information  they  are  designed  to  process  and  use. 

Some  ATRs  use  information  from  only  one  sensor  or 
information  source.  Others  use  information  from 
several  diverse  sources,  fused  at  some  point  in  the 
processing. 


As  predicted,  additional  sources  of  information  which 
permit  new  mission  capabilities  have  been  developed 
for  the  military  cockpit.  For  example,  with  new 
techniques  and  sensors  we  now  envision  attack  at  low 
altitudes  and  high  speeds,  in  all  weather  and  lighting 
conditions.  However  the  same  problems  apparent  in 
1982  still  exist  today:  operator  workload  is  too  heavy 
to  make  optimal  use  of  all  of  this  information  and 
equipment. 


Ultimately  fully  autonomous  ATRs  are  anticipated  that 
perform  all  aspects  of  targeting  from  detection  to 
weapon  release,  without  a  human  operator.  However, 
most  systems  under  current  development  are  not  yet 
robust  enough  to  operate  autonomously  under  all  Field 
conditions.  ATRs  that  detect  well  also  produce  many 
false  alarms;  systems  that  classify  or  identify  make 
many  mistakes  when  they  are  operated  in  field 
conditions  outside  the  laboratory.  ATRs  that  work 
well  in  one  type  of  terrain  or  atmospheric  condition 
may  not  work  well  in  another.  For  at  least  the  next  few 
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yean,  ATRs  an  going  to  need  a  human  operator  who 
can  verify  the  output  of  the  ATR,  who  can  choose 
among  alternatives  reported  by  the  ATR,  or  who  can 
direct  the  ATR  to  process  or  reprocess  sensor  data  of 
interest. 

For  now,  ATRs  can  be  viewed  as  operator  decision 
support  systems:  systems  that  provide 
recommendations  to  the  pilot,  who  retains  the 
responsibility  and  authority  for  making  the  targeting 
decisions.  Viewed  as  a  decision  support  system,  the 
ATR  becomes  another  source  of  information  in  the 
avionics  suite  for  the  pilot.  Its  output  must  be  accepted 
or  rejected  according  to  its  correspondence  with 
information  from  other  available  sources. 

2.2  Operator-ATR  Interface  Issues 
If  an  ATR  implies  "aided"  target  recognition,  then  the 
interface  between  the  operator  and  the  ATR  is 
important  to  the  ease  with  which  the  ATR 
recommendations  are  utilized  and  resolved  with  other 
information.  The  first  element  that  must  be  considered 
in  this  interface  is  the  nature  of  the  ATR  itself  and  of 
its  output.  In  a  recent  survey  of  aircrew  opinions  about 
ATRs  (Ref.  2),  pilots  report  that,  in  addition  to  the 
ATR  recommendation,  they  want  to  know  what  sources 
of  information  the  ATR  has  used  to  make  each  of  its 
recommendations.  They  also  want  available  at  all 
times  a  figure  of  merit  (FOM),  or  quantity  that 
expresses  the  probable  accuracy  of  each 
recommendation. 

Since  the  users  want  more  than  just  minimal  output 
from  an  ATR,  it  is  important  to  examine  the  function 
and  quality  of  the  ATR,  the  nature  of  its  FOM,  and  the 
display  format  for  this  information.  What  will  be  the 
effect  of  providing  all  of  this  ATR  information  on  the 
accuracy  and  timeliness  of  targeting  decisions?  Will 
pilots  be  able  to  resolve  conflicts  that  may  arise  when 
the  ATR  reports  "cruiser"  while  the  Forward  Looking 
Infrared  (FUR)  imagery  shows  a  good  picture  of  a 
frigate? 

The  nature  and  quality  of  other  information  available 
to  the  operator  is  also  important,  as  is  its 
correspondence  to  the  ATR  output.  The  operator  must 
use  this  adjunct  information  to  confirm  or  reject  the 
ATR  recommendations.  Thus  the  quality  of  adjunct 
information  may  affect  the  overall  targeting 
performance  of  the  operator  using  an  ATR. 

In  considering  the  operator  and  ATR  performing  as  a 
system,  it  is  of  great  importance  to  know  whether  the 
information  that  is  available  to  the  operator  was  also 
used  by  the  ATR.  If  both  are  using  the  same  or  similar 
information,  then  their  judgements  wiil  not  be  truly 
independent  of  each  other,  and  both  may  be  based  on 
information  of  similar  quality.  If  the  operator  is  using 
different  information  than  that  used  by  the  ATR,  one 
of  the  two  may  be  using  more  higher  quality,  less 
ambiguous  information  than  the  other.  The  operator- 
ATR  interface  should  reflect  these  variations  and  assist 
the  operator  to  weight  both  the  ATR  output  and  his 
own  judgement  as  a  function  of  the  quality  and 


independence  of  the  information  available  to  each 
source. 

The  decision  making  characteristics  of  the  operator 
must  also  be  reflected  in  the  man-machine  interface. 
Targeting  decisions  are  made  under  time  constraints 
and  under  stress.  A  good  interface  should  mitigate 
operator  errors.  When  ATR  output  disagrees  with  other 
information  sources  such  as  sensor  data  or  sensor 
imagery,  operators  must  resolve  these  differences  by 
integrating  several  probabilistic  and  uncertain 
alternatives.  When  combining  probabilistic 
information,  humans  typically  perform  in  a  sub- 
optimal  manner. 

Human  decision  making  also  is  influenced  by 
systematic  biases.  Anchoring  is  just  one  of  several 
such  biases  that  might  affect  targeting  decisions  made 
by  an  aircrew  using  an  ATR.  Anchoring  in  this  context 
means  that  in  a  situation  where  the  operator  must 
choose  between  conflicting  alternatives,  he  would  be 
unduly  influenced  by  the  information  source  that  he 
first  considers.  The  operator  who  first  notes  the  ATR 
output  would  most  likely  concur  with  the  ATR;  the 
operator  who  first  notes  a  cockpit  imaging  sensor 
would  most  likely  follow  his  initial  interpretation  of 
that  image. 

An  adequate  investigation  of  the  interface  between  an 
operator  and  an  ATR  must  include:  (1)  consideration  of 
the  nature  of  the  ATR  output,  (2)  the  ATR  display 
format  and  accuracy,  (3)  the  additional  information 
sources  available  to  the  operator,  along  with  their 
quality  and  relationship  to  the  ATR  input,  and  (4)  the 
decision  making  characteristics  of  the  operator.  All  of 
these  factors  are  parameters  of  the  interface  between 
the  operator  and  the  ATR  and  must  be  considered, 
varied,  or  controlled  in  an  evaluation  of  the  operator- 
ATR  interface.  To  date  there  has  been  little  research  on 
this  interface.  The  remainder  of  this  paper  describes 
the  design  and  outcome  of  experiments  conducted 
recently  to  learn  about  the  operator-ATR  interface. 

3.  EXPERIMENT  1 

3.1  Method  and  Design 
An  initial  study  (Ref.  3)  compared  the  accuracy  and 
response  times  (RT)  of  an  operator  using  a  simulated 
ATR  to  identify*  images  of  ships  presented  under  time 
constraints  with  the  accuracy  and  speed  of  an  unaided 
operator  in  the  same  situation.  This  experiment 
replicated  portions  of  an  earlier  study  (Ref.  4). 
Operators  were  provided  with  FLIR  imagery  of  ships  in 
conjunction  with  the  output  of  a  simulated  ATR.  The 
subjects'  performance  in  identifying  the  ships  when 
aided  by  an  ATR  was  compared  to  their  unaided 
performance. 


*The  terms  "recognition"  and  "identification"  in  this 
report  are  used  interchangeably  and  refer  to  the  subjects' 
ability  to  provide  the  correct  name  for  a  given  image  from  the 
set  of  images. 


The  FLIR  imagery  showed  a  white-hot  broadside  image 
of  a  single  ship  taken  from  a  set  of  seven  ships.  The 
FLIR  images  of  each  of  the  seven  ships  were  shown  at 
three  different  distances:  near  (5-7  kilometers  (km)) 
which  showed  very  clear,  good  imagery;  medium  (11- 
13  km)  which  gave  fair  imagery;  and  far  (16-18  km) 
which  was  very  poor,  blurred  imagery. 

Three  simulated  ATRs  were  used  which  differed  from 
each  other  in  the  accuracy  of  the  ship  identifications 
that  they  made.  The  most  accurate  was  correct  90%  of 
the  time,  the  next  was  correct  70%  of  the  time,  and  the 
poorest  was  correct  only  50%  of  the  time.  In  all  cases 
the  output  of  the  ATR  was  the  name  of  one  of  the  seven 
ships  and  a  figure  of  merit  (50,  70,  or  90)  which 
expressed  the  overall  accuracy  of  the  ATR.  No 
information  was  provided  to  the  operators  on  how  the 
ATRs  obtained  their  output,  but  in  all  cases  die  quality 
of  the  ATR  output  was  independent  of  the  quality  of  the 
FLIR  imagery  shown  to  the  operators.  Thus  even  when 
the  FLIR  showed  very  poor,  Murry  imagery,  the  90% 
ATR  was  still  correct  90%  of  the  time.  Conversely, 
when  the  FUR  was  excellent,  the  50%  ATR  was  correct 
only  half  of  the  time. 

The  FLIR  imagery  and  ATR  recommendations  were 
presented  for  2,  4,  6,  or  8  seconds.  Subjects  could  take 
longer  to  identify  the  ships,  however  they  could  not 
study  the  imagery  or  the  ATR  recommendation  for 
longer  than  the  time  specified  in  the  experimental 
design.  They  were  told  that  while  accuracy  was  more 
important  than  speed,  they  should  make  their 
identification  decisions  as  quickly  as  possible. 

Sixteen  subjects  participated  in  the  experiment.  All 
subjects  were  pre-trained  to  familiarize  them  with  the 
seven  ships,  and  were  familiarized  with  the 
experimental  conditions. 

3.2.  Results  for  Accuracy  of  Identification 

3.2.1  Aided  Versus  Unaided  Performance 
The  results  showed  that  an  operator  was  significantly 
more  accurate  in  identifying  the  ships  when  aided  by 
an  ATR  than  he  was  when  unaided.  Fig.  1  shows  the 
percentage  of  correct  identifications  for  the  unaided 
operator  and  for  the  operator  aided  by  the  three  ATRs. 
Percentages  of  correct  identifications  increased  both 
as  the  reliability  of  the  ATR  increased,  and  also  as  the 
quality  of  the  imagery  increased  (F(3,45)  »  21.52,  p  < 
0.001  and  F(2,30)  *  49.27,  p  «  0.001,  respectively). 
Fig.  2  shows  the  difference  scores  between  aided 
operator  performance  for  each  of  the  ATRs  and  unaided 
operator  performance:  difference  scores  above  the  zero 
line  mean  that  the  performance  of  the  operator  using 
an  ATR  was  better  than  the  unaided  operator 
performance,  while  those  below  the  zero  line  indicate 
the  opposite. 

Fig.  2  shows  that  in  almost  every  case  the  aided 
performance  was  substantially  better  than  the  unaided. 
Gains  exceeding  15%  were  found  when  operators  were 
aided  by  reliaMe  ATRs  and  when  they  had  imagery  of 
13  km  or  closer.  Even  the  very  poor  50%  ATR,  when 


used  by  an  operator  with  good  or  even  fair  imagery, 
produced  gains  between  4%  and  10%.  Contrasts 
showed  that  all  of  these  gains  in  percentage  of  correct 
identification  were  significant  except  the  overall 
difference  (ignoring  image  quality)  between  unaided 
performance  and  performance  aided  with  ATR  FOM  50. 

3.2.2  Aided  Versus  Autonomous  ATR  Performance 
To  compare  aided  operator  performance  to  the 
performance  of  the  ATR  alone,  difference  scores  were 
computed  by  subtracting  the  percentages  of  correct 
identifications  made  by  each  ATR  acting 
autonomously  from  the  percentage  of  correct 
identifications  made  by  the  operator  using  that  ATR. 
Fig.  3  shows  these  difference  scores. 

As  before,  positive  difference  scores  indicate  that  the 
performance  of  the  operator  using  the  ATR  was  better 
than  the  autonomous  ATR,  while  negative  scores 
indicate  the  opposite.  The  difference  scores  were  not 
different  from  zero  when  averaged  over  all  conditions, 
but  it  is  clear  that  there  was  tremendous  variation  in 
both  the  magnitude  and  direction  of  the  scores 
depending  on  the  ATR  FOM  and  the  FLIR  imagery 
(F(2,30)  =*  14.88,  p  <  0.001;  and  F(2,30)  =  22.43,  p  < 
0.001,  respectively). 

An  operator  using  the  90%  ATR  was  less  accurate  than 
the  autonomous  90%  ATR,  as  shown  by  the  negative 
difference  scores,  while  an  operator  using  the  50% 
ATR  was  almost  always  more  accurate  than  the 
autonomous  50%  ATR.  Contrasts  showed  that  both  of 
these  differences  were  significant. 

Operators  using  the  70%  ATR  were  sometimes  better 
and  sometimes  worse  than  the  ATR  alone,  depending 
on  the  imagery.  On  the  average,  the  contrasts  showed 
that  there  was  no  significant  difference  between  the 
operator  using  the  70%  ATR  and  the  autonomous  70% 
ATR. 

An  analysis  of  operator  compliance  with  the 
recommendations  provided  by  the  ATRs  showed  that 
operators  tended  to  comply  more  with  the  more 
reliable  ATRs,  and  that  their  degree  of  compliance  was 
not  dependent  on  the  quality  of  the  imagery.  Analysis 
also  showed  that,  when  operators  disagreed  with  the 
90%  ATR,  they  almost  always  were  incorrect.  When 
they  disagreed  with  the  less  reliable  ATRs  they  were 
able  to  improve  overall  performance. 

3.2.3  Time-Limited  Information  Presentation 
Identification  accuracy  did  not  vary  as  a  function  of 
whether  the  information  and  imagery  was  presented  for 
2,  4,  6,  or  8  seconds.  These  results  are  shown  in  Fig. 

4.  Although  some  subjects  reported  that  they  would 
have  liked  more  time  in  the  2  second  condition, 
apparently  that  was  sufficient  time  to  identify  the 
ships  and  use  the  ATR  information. 


3.3  Results  for  Response  Time 

RTs  increased  directly  with  exposure  time  (F(3,43)  = 

24.69,  p  <  0.001).  It  is  not  surprising  that  subjects 
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who  had  more  time  to  look  at  the  imagery  took  longer 
to  respond.  These  results  are  shown  in  Fig.  5. 

In  the  RT  analyses  it  is  interesting  that  the  variation 
in  RT  as  a  function  of  the  degree  of  aid  was  marginally 
significant  (F(3,42)  *  2.41,  p  <  0.08).  Fig.  6  shows 
that  the  operator  use  of  a  highly  reliable  (90%)  ATR 
took  less  time  than  the  unaided  decision,  but  operator 
use  of  the  less  reliable  (50%)  ATR  took  longer.  It 
takes  an  operator  time  to  use  an  unreliable  ATR;  the 
operator  can  work  faster  unaided.  Only  good,  reliable 
ATRs  save  operator  time.  Contrasts  show  that  the  RT 
associated  with  the  operator  using  the  ATR  with  FOM 
90  was  significantly  shorter  than  the  RT  associated 
with  the  use  of  the  ATR  FOM  50  (F(l,14)  x  11.56,  p  < 
0.004). 

There  were  no  other  significant  main  effects  or 
interactions  in  the  RT  analyses.  RT  did  not  vary  with 
image  quality  of  the  FLIR.  This  indicates  that  the 
operators  did  not  spend  a  longer  time  in  making  their 
identification  decisions  when  they  had  to  base  their 
decisions  on  poor  imagery  viewed  for  limited,  fixed 
amounts  of  time. 

These  results  are  very  similar  to  the  results  found  in  an 
earlier  experiment  with  two  exceptions  (Ref.  4).  First, 
the  current  subjects  performed  slightly  less  well  in 
almost  every  condition.  Possibly  this  performance 
decrement  was  a  trade  off  of  speed  for  accuracy  since  in 
the  earlier  experiment  there  were  no  time  constraints, 
and  subjects  frequently  spent  the  greater  part  of  a 
minute  examining  the  imagery.  Second,  in  the  earlier 
experiment,  unlike  this  one,  operators  using  the  70% 
ATR  were  consistently  better  than  the  autonomous 
70%  ATR,  regardless  of  the  imagery  presented.  Other 
differences  between  the  results  of  the  two 
investigations  were  minor,  or  could  be  traced  to  the 
differences  just  noted. 

3.4  Discussion 

These  results  have  major  design  implications  for  the 
operator-ATR  interface.  Operator  performance  will 
vary  both  with  the  level  of  accuracy  of  the  ATR  and  the 
quality  of  the  imagery  that  the  operator  has  to  work 
with.  Because  all  of  the  comparisons  showed  that  aided 
performance  was  superior  to  unaided,  it  is  evident  that 
performance  of  the  overall  system  will  be  maximized 
by  using  an  ATR  even  if  it  is  a  very  poor  one,  at  least 
within  the  ATR  performance  ranges  that  were  tested. 

Further  research  is  needed  to  examine  whether  ATRs 
that  are  less  accurate  than  50%  will  also  enhance 
operator  targeting  decisions.  There  may  be  a  lower 
limit  of  accuracy  for  an  ATR  where  the  operators' 
performance  will  no  longer  be  enhanced  by  having 
ATR  information  available.  Another  experiment  (Ref. 
5)  has  shown  that  operators  using  two  sources  of 
imaging  information  are  unable  to  ignore  poor  quality 
information.  They  integrate  poor  and  good 
information,  consequently  doing  less  well  than  they 
would  have  done  had  they  based  their  decision  on  only 
one  good  information  source. 


The  results  of  this  experiment  also  imply  that  an  ATR 
that  is  as  good  as  the  simulated  90%  ATR  should  be 
used  as  an  autonomous  system.  The  operators  did  not 
improve  on  the  performance  of  the  autonomous  90% 
ATR,  regardless  of  the  quality  of  the  imagery  that  the 
operator  was  shown. 

It  is  also  clear  that  the  50%  ATR  should  not  be  used  as 
an  autonomous  system,  rather  it  should  be  used  with  an 
operator.  The  performance  of  operators  using  the  50% 
ATR  was  always  equal  to  or  better  than  the  autonomous 
50%  ATR,  and  also  always  better  than  or  equal  to 
unaided  performance.  In  fact,  with  good  imagery 
(imagery  of  objects  closer  than  13  km)  the  operator- 
50%  ATR  performance  was  synergistic.  That  is,  the 
system  performance  was  superior  to  the  performance 
of  either  component  measured  by  itself.  This 
synergism  was  also  seen  in  the  prior  experiment  (Ref. 
4)  for  operators  using  both  the  50%  and  70%  ATRs. 

The  design  implications  of  performance  using  the  70% 
ATR  ate  less  dear.  If  the  operator  has  good  imagery 
(doser  than  13  km)  plus  the  70%  ATR  output, 
performance  is  equal  to  or  superi.  -  than  that  of  the 
autonomous  ATR  or  of  the  unaided  operator. 
Performance  was  not  always  synergistic,  but  it  was 
always  at  least  equal  to  the  performance  of  the 
component  parts.  With  poor  imagery  however  the 
performance  of  the  operator  using  the  70%  ATR  was 
poorer  than  that  of  the  autonomous  ATR.  This 
suggests  that,  if  this  ATR  were  to  be  implemented  for 
use,  the  ATR  should  be  used  with  an  operator,  but  the 
operator  should  not  be  shown  poor  imagery. 

4.  EXPERIMENT  2 

4.1  Method  and  Design 

In  the  second  set  of  experiments,  the  same  incoming 
imaging  information  was  provided  to  both  the  ATR 
and  the  operator.  The  operator  saw  an  image  of  a  ship, 
and  the  ATR,  using  the  same  type  of  imagery,  produced 
an  identification  of  the  ship. 

The  imagery  was  simulated.  To  create  the  basic, 
undistorted  set  of  simulated  imagery,  15  images  of 
ships,  taken  from  Jane's  Fighting  Ships  (Ref.  6),  were 
constructed  by  representing  broadside  views  of  each 
ship  by  60  vertical  bars.  Each  basic  undistorted  ship 
looked  like  a  sort  of  histogram  (see  Fig.  7  for  an 
example).  These  fifieen  ideal  ship  images  were  given 
to  the  operators  in  hard  copy,  and  in  addition  were 
stored  in  the  computer  as  a  reference  library  set  to  be 
used  by  the  ATR  for  comparison  with  incoming 
imagery. 

The  simulated  incoming  imagery  was  constructed  in 
the  same  way  as  the  ideal  basic  set,  but  to  represent 
sensor  and  environmental  noise,  each  vertical  bar  of 
each  image  was  perturbed  vertically  by  adding  or 
subtracting  values  randomly  selected  from  a  normal 
distribution  with  a  mean  of  zero  and  a  standard 
deviation  varied  according  to  the  amount  of  distortion 
required  in  the  experiment.  Image  quality  could  be 
varied  by  perturbing  each  of  the  60  bars  which 
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constituted  the  image.  This,  in  turn,  caused  variations 
in  the  accuracy  of  the  operator  performance  and  the 
accuracy  of  the  ATR  performance. 

As  in  the  earlier  experiment,  this  experiment  used  two 
distortion  conditions.  The  standard  deviations  of  the 
perturbation  distributions  were  selected  based  on 
pretests  so  that  unaided  operator  performance  and 
autonomous  ATR  performance  would  be  roughly  equal, 
and  so  that  performance  under  High  Distortion  would 
be  about  40%  and  under  Low  Distortion  would  be  about 
80%.  (In  the  experiment  these  percentages  actually 
were  40  and  41%  respectively  for  the  operator  and  ATR 
in  the  High  Distortion  condition,  and  85%  for  the  ATR 
and  54%  for  the  operator  in  the  Low  Distortion 
condition). 

The  recommendation  from  the  ATR  was  presented  in 
several  different  formats.  First,  the  number  of 
recommendations  from  the  ATR  was  varied  in  two 
ways  by  giving  either  the  single,  best 
recommendation,  or  by  reporting  the  best  five 
matches  of  the  incoming  imagery  to  the  stored  library 
images.  Second,  the  FOM  was  presented  in  three  ways: 
quantitatively  (a  number  from  0  to  100);  qualitatively 
(an  adjective,  such  as  "High",  "Medium",  "Low",  or 
"Poor"),  or  not  given  at  all.  An  unaided  condition  also 
was  used  as  a  control. 

Overall,  ATR  recommendations  were  presented  in  six 
different  formats  in  two  distortion  conditions  to 
operators  who  identified  15  ships  using  the  ATR 
recommendations  along  with  simulated  imagery.  A 
seventh  control  condition  determined  unaided  operator 
performance  in  identification  of  these  same  ship 
images.  Altogether  a  total  of  28  subjects  from  two 
different  populations  were  tested  at  two  separate  times 
using  this  design. 

4.2  Results 

Analyses  of  the  results  of  this  experiment  are  still 
underway.  Only  a  few  significant  findings,  common  to 
both  populations  that  were  sampled  in  the  experiment, 
are  highlighted  here.  Operators  using  an  ATR 
performed  significantly  more  accurately  than  they 
performed  unaided,  in  both  the  High  and  Low 
Distortion  conditions.  Unaided  performance  in  the 
Low  Distortion  condition  averaged  54%  for  the  two 
samples,  while  aided  performance  averaged  83%. 
Comparable  figures  in  the  High  Distortion  condition 
averaged  39.7%  and  53.4%.  Performance  in  the  High 
Distortion  condition  was  synergistic:  the  53% 
accuracy  of  the  operator  using  the  ATR  was  also 
significantly  superior  to  that  of  the  autonomous  ATR 
at  40%.  There  was  no  synergism  in  the  Low  Distortion 
condition:  the  operator-ATR  system  performed  at 
83%,  which  was  not  significantly  different  than  the 
autonomous  ATR  performance  of  85%. 

Aiding  reduced  RTs  in  the  Low  Distortion  condition 
from  an  average  of  15.0  seconds  to  11.5  seconds  for 
the  two  experiments.  In  the  High  Distortion 
condition,  however,  aided  RTs  were  not  shorter  than 


unaided  RTs  (respectively  17.4  seconds  and  16.9 
seconds). 

Analyses  of  the  Number  of  Recommendations  and  the 
FOM  Format  conditions  showed  in  both  samples  that 
neither  factor  was  significantly  related  to  performance 
accuracy  in  the  High  Distortion  condition.  In  the  Low 
Distortion  condition,  however,  accuracy  of 
performance  varied  significantly  as  a  function  of  these 
conditions. 

Two  kinds  of  display  formats  resulted  in  the  most 
accurate  performance.  One  of  these  provided  five 
recommendations  along  with  a  quantitative  FOM 
display;  subjects  were  correct  72%  of  the  time.  The 
other  gave  one  recommendation  and  a  qualitative 
FOM;  subjects  were  correct  over  70%  of  the  time. 

The  format  that  gave  five  recommendations  with  No 
FOM  resulted  in  the  worst  performance  with  only  63% 
correct  identifications.  Also  RTs  increased 
significantly  as  a  function  of  the  number  of 
recommendations  and  generally  responses  took 
approximately  2  seconds  longer  when  the  ATR  gave 
five  recommendations  rather  than  one. 

4.3  Discussion 

The  full  implications  of  these  findings  for  design  must 
await  a  complete  analysis.  However  it  is  clear  at  this 
time  that  ATR  display  design  will  affect  the  overall 
performance  accuracy  of  the  operator  who  uses  it.  The 
presentation  of  more  information  from  an  ATR  (such 
as  a  larger  number  of  alternative  identifications  or  the 
presentation  of  FOMs)  does  not  improve  performance 
in  all  cases,  and  it  may  increase  the  response  time. 

The  significant  interaction  between  the  display 
treatments  and  distortion  level  may  indicate  that  the 
optimum  display  format  will  vary  depending  on  either 
the  absolute  quality  of  the  ATR,  or  on  the  quality  of 
the  ATR  relative  to  the  performance  of  the  operator. 
With  an  ATR  performing  about  equal  to  the  operator  at 
fairly  low  40%  levels,  format  does  not  greatly  affect 
accuracy.  A  format  providing  the  single  best  choice 
from  the  ATR,  given  without  a  FOM,  would  yield  the 
most  rapid  response  and  presumably  would  yield  a 
performance  as  accurate  as  with  any  other  format. 

Given  an  ATR  that  is  accurate  more  than  80%  of  the 
time,  one  that  is  about  30%  better  than  the  operator, 
the  optimum  format  might  well  be  the  five  best 
matches  to  the  library  images.  However,  this  is  true 
only  if  quantitative  FOMs  can  also  be  provided  and 
longer  response  times  can  be  tolerated.  Otherwise, 
display  of  five  recommendations  is  a  poor  choice, 
according  to  these  preliminary  analyses. 

In  this  study,  optimum  performance  was  obtained  from 
various  combinations  of  operator  and  ATR 
components.  With  the  very  good  performance  of  the 
ATR  in  the  Low  Distortion  condition,  the  presence  of 
the  operator  did  not  improve  or  degrade  the  accuracy  of 
the  system.  The  ATR  could  act  autonomously  or  not, 
based  perhaps  on  the  workload  imposed  upon  the 
operator  by  other  flight  duties.  However,  with  poorer 
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imagery,  the  operator’s  interaction  with  the  ATR  was 
required  for  optimum  performance.  In  fact,  the  operator 
using  the  ATR  produced  a  synergistic  increase  in 
performance. 

5.  CONCLUSIONS 

Further  experimentation  is  required  to  establish 
guidelines  for  optimum  interfaces  between  operators 
and  ATRs.  The  two  experiments  discussed  here  dearly 
demonstrate  that  the  interface  should  be  varied  as  a 
function  of  the  accuracy  of  the  ATR,  perhaps  measured 
relative  to  operator  performance.  Adjustments  to  the 
interface  could  potentially  be  made  dynamically,  with 
the  degree  of  autonomy  for  both  the  operator  and  the 
ATR  adapted  as  a  function  of  the  relative  performance 
of  each,  the  quality  of  incoming  information,  and 
concomitant  workload. 

The  experiments  also  show  that  the  format  used  to 
display  ATR  information  can  be  adjusted  to  optimize 
performance.  Further  experiments  are  underway  to 
define  additional  parameters  which  will  optimize 
operator-ATR  performance  and  which  will  additionally 
darify  the  multiple  relationships  between  the  target, 
the  sensors),  the  ATR,  the  display,  the  operator,  and 
the  dedsion  making  strategy. 
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Fig.  1.  Identification  Accuracy  as  a  Function  of  the  Interaction  of  Degree  of  Aid  and  Distance. 
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Fig.  2.  Aided  Minus  Unaided  Performance  as  a  Function  of  FOM  and  Distance. 
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Fig.  3.  Operator/ATR  Performance  Versus  ATR  Alone. 
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Fig.  6.  Response  Time  as  a  Function  of  Aid  Level. 


Fig.  7.  CGN-36,  the  California ,  a  Sample  of  the  Simulated  Imagery  Used  in  Experiment  2. 
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Discussion 


QUESTION  E.J.  LOVESEY 

Did  you  use  pilots  for  your  trials?  Was  there  an  additional  task  (e.g. 
flying)  other  than  the  recognition  task? 

REPLY 

Our  subjects  were  scientists  and  engineers  employed  at  NWC .  On  other, 
similar,  target  recognition  experiments  we  have  used  pilots  as  well  and 
have  not  found  any  systematic  differences  between  them  and  the  non-pilots 
in  terms  of  the  accuracy  of  their  identifications. 

We  did  not  use  a  secondary  task.  We  felt  that  the  secondary  tasks 
generally  used  did  not  properly  represent  the  kinds  of  tasks  that  occur 
during  the  targeting  phase  of  a  mission. 
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Summary 

After  a  protracted  development  period,  the  active- 
matrix  addressed  liquid  crystal  light  valve  is  finding 
increasing  employment  in  a  variety  of  head-down 
primary  flight  instrument  applications.  Whilst  the 
packaging  efficiencies  of  such  a  flat  panel  technology 
are  self-evident,  many  of  its  other  attributes  are  also 
likely  to  have  a  profound  impact  upon  crew  station 
design.  In  order  to  successfully  exploit  the  positive 
aspects  of  this  technology  it  is  appropriate  that  the 
avionics  systems  architects  benefit  from  the,  albeit 
limited,  current  operational  data  base  and  develop  a 
realistic  understanding  as  to  the  limits  and  rate  of 
advancement  pertinent  to  this  class  of  display  device. 
The  following  paper  is  offered  as  an  element  in  this 
information  dissemination  process  from  a  source  that 
has  maintained  a  presence  in  the  technology  since 
1984  and  has  participated  in  AM-LCD  solutions  for 
such  diverse  air- vehicle  requirements  as  YF-23,  LH, 
P-3CandC-130 

The  content  comprises: 

(a)  A  cursory  understanding  of  fundamentals. 

(b)  An  overview  of  nature  and  extent  of  current 
operational  examples  of  this  technology  as 
known  in  the  public  domain. 

(c)  An  outline  of  specific  primary  flight 
instrument  usage  based  upon  a  rigorous  C-130 
flight  test  program. 

(d)  A  discussion  of  areas  of  ongoing  research  and 
advancement  in  terms  of: 


(i)  Growth  in  display  area/resolution/ 
modulation  depths. 

(ii)  Centralized  and  distributed  processor 
architectures. 

(iii)  Photometric  efficiency  via  novel  colour 
generation  and  lighting  methods. 

iv)  Incorporation  of  autostereoscopic  feat¬ 
ures  resultant  from  the  discrete  pixel 
structures  of  the  light  valve  device. 

1.  The  Concept 

The  multiplexed,  matrix-addressable  liquid-crystal 
(LC)  cell  operated  in  a  transmissive  (back-lit)  made 
has  offered  itself  as  a  potential  alternative  to  CRT 
devices  in  a  variety  of  display  applications  for  some 
considerable  period.  Despite  spectacular  advances  in 
the  non-linear  electro-optic  properties  of  the  basic 
mesomorphs  and  similar  sophistication  in  cell 
structures,  the  multiplexing  level  for  simple  cell 
structures  has  not  approached  the  level  satisfactory 
for  large  area,  high  information  content/video 
capable  displays  suitable  for  the  airborne 
environment.  This  restriction  in  performance  of 
simple  LC  structures  has  been  well  covered  in  prior 
works  and  continues  to  be  a  focus  of  effort  at  the 
basic  materials/process  level  as  evidenced  by 
advanced  nematic  eutectics,  super-birefringent 
structures  and  the  optically  bi-stable  ferro-electrics. 
Notwithstanding,  the  Display  Industry  has  sought  a 
more  direct  solution  to  the  high  performance/high 
resolution  flat-panel  conundrum  via  the  pursuit  of 
'active-addressing’  each  element  of  the  viewable 
surface.  In  this  mechanisation,  each  matrix  cell 
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contains  an  additional  non-linear  element  (normally 
diode,  transistor  or  Poole-Frenkel  structure)  in  series 
with  the  LC  cell,  which  increases  the  composite 
multiplexing  level  consistent  the  needs  this  display 
category. 

Due  to  the  substantial  nature  of  the  market  for  such 
a  highly  capable  display  flat  panel  display,  the 
detailed  engineering  solutions  to  such  'active-matrix’ 
mechanisations  are  manifold  and  the  subject  to  much 
debate  as  to  their  'correctness’.  Fortunately,  these 
nuances  are  not  germane  to  our  panel  topic,  thus 
enabling  us  to  more  forward  to  more  relevant  issues 
on  the  understanding  that  a  plurality  of  viable 
technical  solutions  to  the  AM-LCD  approach  have 
been  reduced  to  practice  and  validated. 

Suffice  to  say  the  active- matrix  method  (by  whatever 
approach)  allows  a  multiplex  ratio  of  1000:1  without 
difficulty  thus  permitting  large  area  (~300  cm2) 
display  surfaces  of  resolution  and  refresh  equivalent 
on  better  than  achievable  in  the  shadowmask  CRT 
domain. 

Given  that  such  a  display  solution  is  technically  and 
economically  realizable,  we  must  now  understand 
how  it  may  be  employed  to  best  effect,  become 
mindful  of  its  existing  established  deficiencies, and 
exploit  any  unique  positive  attributes  it  may  possess. 

In  order  to  do  this,  consideration  of  the  generic 
structure  of  the  light  valve  itself  (Figure  1)  and  the 
supportive  elements  of  the  complete  multi-function 
display  (Figure  2)  may  be  useful.  Here  the  light 
valve  employs  a  conventional  structure,  placing  the 
addressing  matrix  on  the  rear  surface  of  the  cell,  the 
absorptive  colour  filter  elements  on  the  opposing 
front  surface,  in  corqunction  with  a  twisted-nematic 
cell  structure  (utilising  crossed  or  parallel 
polariser/analyser  elements)  and  a  bulk  scattering 
diffuser.  Each  of  these  attributes  bounds  defines  or 
the  optical  performance  of  the  cell  (in  regard  to 
transmission  efficiency,  contrast,  chromatic  stability 
and  other  associated  photometric  parameters). 
Beyond  the  cell  itself,  the  immediate  peripheral 
electronics  must  concern  itself  with  addressing  the 
active  array  (in  terms  of  level  shifting,  grey-shade 
coding,  refresh  rate)  in  accordance  with  the  display 


image  memory,  thus  introducing  constraints  in  total 
performance  with  regard  to  image  fidelity  and 
moving  artifacts. 

Complementary  to  the  electro-optic  assembly  is  the 
lighting  module,  required  to  provide  a  high 
conversion  efficiency,  variable  luminous  flux  source, 
spectrally  compatible  with  the  colour  filter 
parameters.  This  element  therefore,  becomes  the 
dominant  influence  in  terms  of  visual  dynamic  range 
and  thermal  management. 

Finally,  the  core  MFD  electronics  provisions  for  the 
creation/acquisition  of  the  displayed  image 
performing  such  functions  as:-  artifact  suppression, 
scaling,  translation  and  overlay/windowing. 

2.0  Implementation 

Current  solutions  to  the  above  technical  challenges 
have  now  reached  the  level  of  maturity  to  permit  a 
variety  of  viable  demonstrations  of  capability. 
Flight  validation  of  such  equipment  has  been 
accomplished  in  diverse  air-vehicles  such  as  P-3C 
(part  of  the  update  IV  configuration),  AH  64  (in 
support  of  LH  (RAH-66)  risk  reduction)  and  YF-22 
(as  an  element  of  ATF  source  selection).  However, 
the  most  vigorous  investigation  of  the  AM-LCD  MFD 
in  the  flight  regime,  is,  in  all  probability,  evidenced 
in  the  continuing  evaluation  being  conducted  as  part 
of  a  larger  RAMTIP  (Reliability  And 
Maintainability  Technology  Insertion  Program) 
C-130  initiative. 

This  C-130  architecture,  as  developed  by  Litton 
Canada,  is  configured  about  6  identical  AM-LCD 
MFDs  serviced  by  dual  6-channel  data 
acquisition/display  processors  over  125  MHz  fibre- 
optic  communications  links.  The  system  provides 
multi-mode  operation  at  each  display  station 
inclusive  of  primary  flight,  navigation,  engine 
instrumentation,  advisory/caution/  warning  and 
weather/ground  map  thereby  offering  a  complete 
emulation  of  current  EFIS  capabilities,  with 
mechanical,  electrical  and  optical  characteristics 
corresponding  to  the  data  as  shown  in  Table  1  Via 
the  usage  of  the  AM-LCD  technique  this  functional 
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equivalence  to  state-of-the-art  CRT  systems  is 
augmented  by  the  following  invaluable  features: 

The  volume/weight  economies  (The  LCD  MFD 
is  6  kg  and  15  cm  deep,  approximately  35%  of 
its  CRT  counterpart)  yield  logistical 
simplicity,  dry-weight  reduction  and  retrofit 
savings  (in  the  C-130  case  the  display  suite 
was  incorporated  without  significant 
structural  alteration  to  the  air-vehicle). 

The  optical  characteristics  of  the  LCD  closely 
parallel  that  of  the  full-colour  CRT  under  most 
general  lighting  conditions  but  also  provide 
significantly  improved  contrast  and  colour 
stability  in  the  high-ambient  sunlight 
environment,  a  condition  more  prevalent  in 
the  theater  transport  "greenhouse”  crew- 
station. 

The  nature  of  the  LCD  device  allows  the  MFD 
to  achieve  an  M.T.B.F.  in  excess  of  5000  hrs, 
some  3-4  times  greater  than  equivalent  CRT 
systems.  Moreover,  the  LCD  structure  is,  in 
essence,  a  massively  parallel  light  valve, 
thereby  ensuring  graceful  (fail  operational) 
degradation  under  most  circumstances. 

The  LCD  MFD  contains  no  elements  requiring 
calibration/harmonization  nor  includes 
obscure  or  high-voltage  electrical  services. 
This  permits  the  establishment  of  a  true  2- 
level  maintenance  philosophy,  which  can,  as 
required,  be  further  condensed  to  an  all-on- 
aircraft  repair  strategy  for  austere 
deployments. 

The  C-130  installation,  as  illustrated  in  Figures  3 
and  4  therefore  has  validated  the  merits  of  LCD 
EFIS  methods  in  a  Theater  Transport  environment 
in  the  following  areas: 

Significant  weight  reduction  ( ~  40  kg/set) 

Enhanced  visual  information  available  to 
crew 

Optimized  air-vehicle  availability/ 
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supportability 

Competitive  acquisition  cost  (as  compared  to 

CRT)  and  reduced  LCC 

Minimized  crew-station  disturbance  for 

retrofit/MLU  usage. 

While  the  above  benefits  may  be  impressive  in 
themselves,  they  represent  only  a  marker  on  a 
rapidly  developing  technology  front.  The  C-130 
system  as  described,  passed  through  its  formative 
engineering  phase  in  1988  and  in  the  ensuing  period 
considerable  further  advance  has  been  realized. 

3.0  Growth 

At  time  of  publication  the  MFDs  utilized  in  the  C- 
130  evaluation  have  yielded  some  15,000  hrs  of 
operational  data,  inclusive  of  more  than  250  hrs  of 
aircraft  flight  test. 

Passing  over  the  essential  success  of  this  project, 
many  valuable  lessons  can  be  extracted  from  this 
data  base  to  the  benefit  of  future  generations  of  this 
technology.  The  most  significant  of  these  rests  in  the 
area  concerning  the  interplay  between  photometric 
performance  and  thermal  management.  The  crew- 
station  under  consideration  was  designed  in 
accordance  with  accepted  Human  Factors  guidelines 
concerning  the  efficient  assimilation  of  colour 
encoded  alpha-numeric  and  graphical  data  when 
viewed  under  the  broad  range  of  airborne  ambient 
lighting  conditions.  Compounded  in  this  was  the 
necessity  to  support  cross-cockpit  viewing  of  such 
data  without  incurring  loss  of  contrast  or  chromatic 
integrity  that  may  impede  safe  flight  operations. 
Utilising  the  accepted  TN  structure  in  conjunction 
with  a  clear  aperture  (active- matrix)  ratio  of  55-58% 
this  places  great  emphasis  upon  the  peak  output  of 
the  backlighting  element  thus  requiring  a  power 
budget  for  this  module  in  the  order  of  1.5W/sq.in.  of 
viewing  surface.  Although  not  excessive  as 
compared  to  an  equivalent  CRT  systems  within  the 
reduced  volume  of  LCD  configuration,  this 
dissipation  requires  the  provision  of  forced  cooling  to 
support  achievement  of  the  desired  reliability  goals. 
Improvement  on  this  front,  toward  either  obtaining  a 
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DISPLAY  UNIT 

OPTICAL  PERFORMANCE 

DISPLAY  AM-LCD  640x480  Colour 

Quad  Plxal  Configuration 
LUMINANCE  140  FL  Whita  Field 
CONTRAST  25:1  Nominal, 

3:1  at  viewing  angle  extremes 
VIEWING  ANGLE  Horizontal:  +/-  60  degrees, 
Vertical :  +30,  <5  degrees 

COLOUR  256  from  a  pallet  of  2046 
GREY  SHADES  16  Levels 
UPDATE  RATE  30  Hz,  Interlaced 
REFRESH  RATE  60  Hz,  Non-Interlaced 

INTERFACES 

FIBRE-OPTIC  Dual  Redundant,  125  MHz  Links 
VIDEO  NTSC  Standard  Video  Output 

SERIAL  Dual  Redundant  RS-422 

High  Speed  Channela 

ENVIRONMENTAL 

Oesigned  to  meet  requirements  of  MIL-E-5400 
Class  1AX. 

RELIABILITY 

MTBF  -  8744  Hours  min.  per  MIL-STD-217E 
parts  count  method. 

MAINTAINABILITY 

No  scheduled  Maintenance  Requirements 
MTTR  •  20  minutes  (MIL-STD-470) 

DIMENSIONS 

6.7F  W,  6.75"  H,  8.(T  D. 

POWER 

28  VOC  @  125  W.  (per  MIL-STD-704) 

WEIGHT 

14  Pounds 


DISPLAY  PROCESSOR 

HARDWARE  DESCRIPTION 

ARINC  429  INTERFACE 
Input  Ports:  2  @  12.5  KHz 
:  4  @  100  KHz 
Output  Ports:  2  @  100  KHz 
1553B  INTERFACE 

2  Dual  Redundant  ports  capable  of  either  Bus 
Controller,  Bus  Monitor  or  Remote  Terminal  Mode 
FIBRE-OPTIC  INTERFACE 

6,  125  MHz  Outputs,  1.3  uM  wavelength 
DISCRETE  INTERFACE 

Input  Discretes:  48  (28  VDC  or  Ground  ref.) 
Output  Discretes:  16  @  40  mA  sink  capability 

ANALOG  INTERFACE 
Input :  16  @  0-5  VDC 

Output:  4  @  26  VAC,  400  Hz 

SYNCRO  INTERFACE 

Input :  16-3  wire  Syncro  Inputs,  @26VAC,  400  Hz 

RADAR  INTERFACE 

Video  I.A.W.  EIA  RS-343A 

SERIAL  INTERFACE 

8  -  EIA  RS-422  Hl-speed  serial  ports 

GRAPHICS  PROCESSORS 
6  Independent  34020  Based  GPU’s 
VME  Bus  Structure 

ENVIRONMENTAL 

Designed  to  meet  requirements  of  MIL-E-5400 
Class  1AX. 

RELIABILITY 

MTBF  -  5082  Hours  min.  per  MIL-STD-217E 
parts  count  method. 

MAINTAINABILITY 

No  scheduled  Maintenance  Requirements 
MTTR  -  30  minutes  (MIL-STD-470) 

DIMENSIONS 

20.26"  L,  10.5"  W,  8.5"  H,  (excluding  mounting  tray) 

POWER 

28  VDC  @  190  W  (per  MIL-STD-704) 

WEIGHT 
48  pounds 


Table  1  C-130  RAMTIP  EFIS  System  Technical  Specification 
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passively  cooled  package  or  higher  display 
luminance  (at  the  same  power  level)  thus  enhancing 
grey-shade  depth/minimizing  chromatic 
desaturation  at  the  higher  ambient  illumination 
levels  would  seem  meritted  and  practical. 

Opportunities  for  advancement  in  this  regard  can  be 
considered  on  four  fronts:- 

Backlight/Defuser. 

The  existing  hot-cathode  fluorescent  devices  are 
probably  difficult  to  significantly  improve  upon  as 
efficient  large  area  sources  for  this  application. 
However,  substantial  growth  in  composite 
performance  is  now  realizable  by  the  more  effective 
collection  and  distribution  of  the  tube  emissions. 
Early  devices  utilised  parabolic  on  planar  reflectors 
in  conjunction  with  bulk  dispersive  diffusers.  Recent 
developments  in  terms  of  linear  involute  collectors 
and  volume  hologram  forward  scatterers  correctly 
applied  offer  some  25-30%  improvement  in  efficiency 
for  this  element  of  the  MFD. 

Matrix  Clear  Aperture. 

Utilising  6-8pm  design  rules  which  appear  to  be  an 
economic  limit  for  the  construction  of  large  area 
active-matrix  arrays  creates  considerable  loss  of 
functional  pixel  area  due  to  the  necessary 
tolerancing  between  source/gate  bus  structures  and 
the  associated  LC  pixel  electrode.  Any  reduction  in 
such  tolerances  in  a  conventional  design  has  the 
potential  to  increase  undesirable  coupling  between 
buses  and  pixel  structures  (create  unacceptable 
artifacts)  or  in  the  worst  case  produce  hard  pixel 
defects  rendering  the  device  unusable.  Via  advances 
in  process  methodology  and  photolithographic 
techniques,  it  is  now  practical  to  'self-align’  the 
actual  pixel  utilizing  the  source/gate  bus  structures 
to  form  part  of  the  exposure  mask.  These  processes 
when  applied  to  a  matrix  having  cell  dimensions  in 
the  order  of  0.15  X  0.15  mm  can  increase  the 
aperture  from  the  previous  55/58%  to  approximately 
70%,  equivalent  to  an  improvement  in  total  matrix 
transmission  by  20  -  28%. 


Cell  Structure/Optical  Enhancement. 

Existing  TN  structures  have  served  well  in  the 
development  of  this  class  of  display.  However  the 
requirement  to  include  dual  linear  polarisers  places 
an  upper  limit  upon  the  cell  transmission  and  the 
birefringent  nature  of  the  structure  invokes 
observable  chromatic  variation  within  the  viewing 
cone  of  display  device.  Opportunities  for 
improvement  in  these  characteristics  are  now 
available  both  by  further  TN  cell  refinement  and  via 
substitution  with  alternate  cell  constructs. 

Within  the  mainstream  TN  domain,  the  perfection  of 
superfluorinated  eutectics  and  high  efficiency 
polarising  materials  now  permits  the  fabrication  of 
cells  of  near  minimum  parasitic  loss.  However, 
associated  approaches  for  the  recovery  and  re-use  of 
the  rejected  polarisation  state  have  to  date  not 
proved  practical  within  the  confines  of  the  MFD, 
although  this  work  is  ongoing.  In  terms  of  chromatic 
stability  and  associated  off-axis  contrast,  two 
mechanisms  have  been  shown  to  be  practical  and 
useful.  Firstly  the  addition  of  low  concentration, 
high-alignment  dyes  to  the  TN  mixtures  has  been 
demonstrated  as  effective  in  reducing  chromatic  shift 
and  broadening  the  iso-contrast  characteristic. 
Secondly,  the  incorporation  of  optical  retardation 
films  whose  principal  axes  bear  specific  relationship 
to  the  cell  alignment  direction  have  proven  effective 
in  stabilising  wave-length  dependent  elliplicity  of 
the  polarisation  state  of  the  transmitted  light  which 
forms  the  underlying  cause  of  oblique  angle 
chromatic  shift. 

Beyond  the  TN  method,  previously  known  but  less 
developed  techniques  that  do  not  require  the  incident 
light  to  be  of  a  linearly  polarised  state  are  now 
approaching  useful  maturity.  Detailed  comment  on 
these  materials  is  beyond  the  scope  of  this  paper 
however,  progress  in  both  the  Heilmeier  (single 
polariser)  and  Suspended  Particle  areas  is  relevant 
in  this  regard. 

Colour  Generation. 

As  shown  in  Figure  1,  the  conventional  method  for 
obtaining  full  colour  performance  from  this  class  of 
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display  is  to  employ  a  front  surface,  mosaic  colour 
filter  registered  to  the  matrix  structure  and 
possessing  transmission  spectra  matched  to  the 
emission  bands  of  the  back-light.  Although 
functional  this  approach  carries  with  it  some 
significant  penalties,  in  terms  of  total  light  loss  (due 
to  absorptive  nature  of  filters)  and  in  the  tendency  to 
utilize  broad  transmission  spectra  (in  order  to 
minimize  light  loss)  which  results  in  a  relatively  de- 
saturated  colour  gamut.  A  number  of  approaches 
have  been  advanced  and  evaluated  with  the  intent  of 
circumventing  the  absorptive  loss  mechanism.  Of 
these,  the  frame-sequential  method  is  best  known, 
however  this  does  not  easily  adapt  itself  to  the  high 
performance  MFD  due  to  the  requirement  to 
maintain  high  frame  rates  (~200Hz)  which  are 
inconsistent  with  the  current  switching 
characteristics  of  the  LC  cell.  A  more  promising 
solution  utlises  a  luminescent  mosaic  in  place  of  the 
absorptive  structure  in  an  inverted  fashion,  whereby 
each  element  is  excited  to  emit  in  its  chosen  spectra 
via  a  back  illuminant  attuned  to  the  appropriate 
absorption  band.  By  either  mechanism  it  can  be 
shown  that  overall  conversion  efficiency 
(lumens/watt)  may  be  improved  by  a  factor  of  2  as 
compared  to  an  equivalent  multi-colour  absorptive 
system. 

4.0  The  Future. 

At  the  present  time  the  AM-LCD  approach  has 
shown  itself  capable  of  executing  an  effective  visual 
emulation  of  the  accepted  CRT  primary  flight 
instrument.  The  C-130  deployment  referenced 
herein  provides  one  of  the  more  substantial 
demonstrations  of  this  capacity,  however  several 
analogous  developments  have  also  been  reliably 
reported.  In  the  various  Centres  of  Expertise 
technical  improvements,  (as  referenced  in  Section  3), 
continue  to  be  incorporated  which  in  the  immediate 
future  will  render  the  AM-LCD/CRT  visual 
equivalence  debate  passe.  Moreover,  upon  achieving 
this  plateau,  the  unique  positive  attributes  of  this 
technique  will  become  more  obvious  and  open  to 
useful  exploitation. 

At  the  display  surface  level  the  following  topics 
present  themselves  as  relevant:- 


Resolution/Modulation. 

For  the  flight  regime  current  AM-LCD  instruments 
have  shown  themselves  capable  of  working  reliably 
at  resolutions  in  the  order  of  0.15mm/cell  in 
conjunction  with  20-30  discernible  grey-levels. 
Advancement  on  these  fronts  is  keyed  primarily  to 
process  engineering  maturity/characterization  and 
not  to  any  immediate  technical  barrier.  It  is 
therefore  reasonable  to  expect  the  resolution  to  more 
toward  0.08-0.  lOmm/cell  with  grey-level  capacity  of 
150-200  without  difficulty  or  delay.  These 
performance  parameters  will  transcend  our  prior 
knowledge  base  in  terms  of  avionics  CRT 
characteristics  and  therefore  cause  us  to  revisit  some 
of  our  established  cognitive  and  perceptual  ground 
rules.  On  a  more  practical  front  such 
resolution/modulation  flexibility  will  afford  the 
construction  of  display  devices  which  exactly  echo 
the  formats  of  advanced  electro-optic  sensor  arrays, 
thus  mitigating  informational  loss  and  artifact 
generation. 

AreaVForm-Factor. 

As  a  producible/deployable  device  the  AM-LCD 
industry  is  currently  constrained  to  devices  in  the 
400cm2  range  containing  1-1. 5M  addressable 
elements.  Extension  of  these  boundaries  will  be 
contingent  upon:- 

Development  and  maturity  of  cost-effective 
process  equipments  suitable  large  area 
substrates. 

Technological  advancement  at  semi-conductor 
device  level  to  satisfy  the  more  stringent 
parametric  requirements  of  the  larger 
area/denser  arrays. 

These  issues  are  non-trivial  however  both  industry 
momentum  and  existing  scientific  knowledge  at  the 
device  level,  suggest  that  the  1200  cm2  device 
containing  3-4  M  elements  should  become  a  usable 
commodity  at  the  avionics  level  within  the  next  2-3 
years. 

Additionally  this  growth  should  have  two  effects  at 
the  MMI  level:- 
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Firstly  we  will  start  to  realise  in  flyable 
configuration  the  more  visually  integrated  crew- 
station  that  has  received  substantial  laboratory 
consideration  to  date.  Secondly,  the  push  forward  in 
area/density  will  also  create  the  capability  and 
knowledge  to  more  economically  and  flexibly 
engineer/produce  devices  of  more  modest 
proportions.  That  is,  the  crew-station  designer  and 
human-factors  engineer  with  the  afforded  to 
opportunity  to  tailor  the  viewing  surfaces  of  each 
requirement  in  an  optimal  fashion  without 
prohibitive  cost. 

Secondly,  at  the  architectural  level  the  volume 
efficiencies  (here  today)  and  power  economies 
(coming)  of  the  AM-LCD  provide  great  flexibility  in 
incorporating  truly  distributed  flight  information 
systems  in  future  air-vehicles.  Discussion  of  the 
merits  of  such  techniques  in  terms  of  LCC,  fault 
tolerance,  damage  containment  and  logistics  as 
compared  to  centralised  systems  are  beyond  this 
paper’s  scope  however,  it  should  be  noted  that  AM- 
LCD  is  the  enabling  technology  for  consideration  of 
these  architectures  without  functional  compromise. 

Finally  while  we  continue,  in  the  main,  to  employ 
the  AM-LCD  to  emulate  emmisive  devices  such  as 
the  CRT,  its  intrinsic  properties  as  a  discretely 
addressable  spatial  light  modulator  should  not  be 
overlooked.  At  the  crew-station  level  we  continue  to 


evaluate  the  value  of  stereopsis  as  a  flight  aid. 
Whilst  research  evidence  shows  promise  the 
practical  realization  of  such  a  display  mode  for 
general  cockpit  usage  continues  to  be  problematic 
As  has  been  demonstrated,  the  properties  of  AM- 
LCD  (Resolution,  video  compatibility,  discrete 
addressability)  when  used  in  conjunction  with 
appropriate  lighting  constructs  can  provide  an 
autostereoscopic  capability  that  may  prove  suitable 
for  such  deployment. 

5.  Conclusion 

Substantial  global  financial  and  technical 
investment  has  produced  an  AM-LCD  body  of 
knowledge  suitable  for  avionics  usage.  A  small 
number  of  centres  of  expertise  have  successfully 
transitioned  this  technology  into  the  airborne 
environment  and  established  its  practical  value  at 
the  development  level.  As  a  result  we  will  witness 
operational  deployment  of  such  devices  in  a  number 
of  air  vehicles  in  immediate  future.  These  initial 
applications  will  provide  parity  of  MMI  performance 
with  existing  systems  while  offering  the  user 
logistical  on  economic  benefit.  However,  ongoing 
AM-LCD  development  is  and  will  continue  to  enable 
new  classes  and  kinds  of  display  devices  that  will 
have  profound  and  valuable  impact  upon  our  man 
machine  interface  concepts. 
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LES  ECRANS  A  CRISTAUX  LIQUIDES 
REMPLACANT  DU  CRT  ET  CLE  DES  FUTURS  COCKPITS 
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Sommaire : 


Due  to  well-knowed  advantages  such  as  a 
small  weight,  reduce  volume  ,  low 
consumption,  lisibility  (under  high  brightness) 
reliability,  to  mention  the  most  important,  the 
flat  panels  have  already  started  to  replace  the 
CRT  in  the  world  of  military  aircraft. 

Among  flat  panels,  the  liquid  crystal  active 
matrix  display  is  the  most  advanced.  Full 
colors  and  grey  shades  displays  are  mass 
produced  for  commercial  applications,  and 
new  military  cockpits,  both  for  airplanes  or 
helicopters  are  based  on  that  technology 
(Rafale  is  used  as  example  to  illustrate  the 
advantages  of  LCD  compare  to  CRT). 

Furthermore  liquid  crystal  panel  is  a  technical 
key  which  will  help  future  cockpit  concepts  to 
wake-up  to  life.  The  head  level  display  and  the 
large  interactive  display  are  among  them. 
Projection  technics  and  liquid  crystal  cells  are 
merged  to  take  benefits  of  liquid  crystal, 
removing  the  drawbacks  (that  notion  is 
illustrated  by  a  brief  description  of  an  head 
level  display  demonstrator). 


Abrdviations 


LCD  : 
AMLCD  : 

THT  : 
DEMVAL  : 
VTL  : 
TRC 
HLD  : 


Liquid  Crystal  Display 
Active  Matrix 
Liquid  Crystal  Display 
Trks  Haute  Tension 
Demonstration  Validation 
Visu  Tftte  Lateral  e 
Tubes  I  Rayons  Cathodiques 
Head  Level  Display 


i.g£n£raut£s 

Les  proprietds  des  cristaux  liquides  sont 
connues  et  utilisdes  de  longue  date  et  leurs 
applications  sont  apparues  dans  le  domaine  de 


la  visualisation  avionique  depuis  quelques 
amides. 

Parmi  les  autres  technologies  d’dcrans  plats, 
(plasma,  electroluminescent,  diodes  ou  tubes  k 
rayons  cathodiques  plats)  seules  les  matrices 
actives  k  cristaux  liquides  concurrencent  les 
dcrans  k  tubes  classiques  dans  le  domaine 
avionique. 

La  couleur  est  le  probldme  le  plus  complexe 
rencontrd  par  les  autres  t eh nologies (plasma, 
electroluminescent)  bien  que  quelques 
realisations  en  laboratoire  laissent  parattre 
quelques  possibilites. 

Cependant  c’est  1’aptitude  k  l’industrialisation 
qui  fait  que  les  cristaux  liquides  sont 
aujourd’hui  en  production  de  masse  (dcran 
pour  PC  portable,  notebook,  cartography  pour 
voiture)  et  totalement  credible  pour  les 
applications  de  type  avionique  militaire. 

D’oh  les  nombreux  appels  d’offres  requdrant 
des  cockpits  munis  d’dcrans  k  cristaux  liquides 
(F22  amdricain,  C130  ou  C141,  Rafale, 
Hdlicoptfere  Tigre  ou  LH)  surtout  aprks  la 
rdussite  des  essais  en  vol  de  la  version 
DEMVAL  de  1’ATF,  et  du  Rafale. 

Dans  la  majoritd  des  cas,  les  dcrans  specifies 
sont  de  type  matrice  active  (AMLCD), 
association  d’une  matrice  de  transistors  ou  de 
diodes  en  silicium  et  du  cristal  liquide  dit 
ndmatique  en  hdlice. 

Ces  dcrans  k  cristaux  liquides  k  matrice  active 
n’dmettent  pas  de  lumidre  mais  la  transmettent, 
dissociant  ainsi  commande  et  puissance 
lumineuse,  permettant  resolution,  piqud 
d’image,  niveaux  de  gris,  sans  prejuger  du 
niveau  de  luminositd. 

2.  LES  VISUALISATIONS  TfcTES  BASSES  LCD 
La  grande  majoritd  des  nouveaux  cockpits 
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TRANSISTOR  CONTROLLED  TWISTED  NEMATIC  LIQUID 


*  2  polar  iseura  lin&ires  places  1’un  dev  am 
I'autre  trantnwctem  ou  absorbent  la  lumitre. 

*  le  criatal  liquide  placd  entre  lea  polameun  a 
au  tepoa  une  structure  en  Mice  qui  modi  fie  la 
polarisation  de  la  lumifcre  (90*).  Sous  l’action 


d’une  tension,  cette  structure  est  dStruite,  la 
polarisation  de  la  iumiire  n’est  pas  modifide. 
*  le  sandwich  (potariaeurs  et  cristal  liquide)  a 
alors  la  propridtd  de  transmettre  ou  d’absoiber 
la  lumibre  suivant  la  tension  appliqude. 


Figure  1  :  Principes  des  dcrans  LCD 


d’avions  ou  d’heiicoptferes  (retrofit  ou 
appareils  neufs)  demandent  des  dcrans  & 
cristaux  liquides.  Les  plus  remar quables 
realisations  dans  le  domaine  militaire  sont 
l’YF22  US  (LOOCKHEED  phase  DEMVAL  - 
Technologic  General  Electric  initiale  -  dcrans 
6.25"x4"  et  6.25"x6.25"),  et  l’avion  Rafale 
(Dassault  -  Technologie  Thomson-LCD 
Evolution  de  ceile  de 

General  Electric  -  dcrans  5"x5").Ainsi  que 
rhglicopt&re  Tigre. 

Nous  allons  ddcrire  cette  demi^re  pour 
iilustrer  notre  propos. 

Dans  le  cadre  du  programme  de  "Visualisation 
Tftte  Lateral e  (VTL)  du  Rafale  D";  les 
specifications  dimensionnelles  de  l’equipement, 
dues  &  la  forme  profile  de  la  cabine,  ont 
impose  la  technologie  LCD  (voir  figure  2.1). 

Fonction  : 

La  visualisation  tfite  laterale  est  une 
visualisation  integree  multimode  couieur  4 
cristaux  liquides  connect  £e  directement  au  bus 
numdrique  avion  (1SS3B).  EUe  intdgre  dans  le 
meme  volume  les  fonctions  d’interface,  de 
calcul  et  de  visualisation  (fonctions  realisdes 
habituellement  en  2  dquipements  :  la  visu  et 


son  boitier  de  calcul). 

Architecture  Electronique  : 

Autour  d’un  dcran  LCD  5"x5",  la  VTL 
regroupe  (voir  figure  2.2)  : 

-  une  visualisation  LCD  et  son  eclairage, 

•  une  electronique  de  generation  d’images 
regroupant : 

.  Ia  fonction  traitement  et  son  interface  avec  ie 
bus  numerique  avion, 

.  la  fonction  generation  d’images  incluant  le 
processeur  graphique,  une  electronique  de 
lissage  de  trace  associee  et  une  interface  de 
traitement  video  STANAG  3350. 

.  une  fonction  interface  de  restitution  de 
mission 

- 1’ alimentation  basse  tension  de  l’equipement 

•  une  interface  de  dialogue  constitude  d’un 
ecran  tactile  et  de  son  contrOleur  associe. 

Performances 
->  L’Optique 

.  Ecran  LCD  S”x5"  haute  resolution  equipe  de 
son  systfcme  d’edairage  arrive. 

Cet  ecran  intdgre  un  syst^me  de  prdchauffage 
permettant  une  mise  en  oeuvre  accdierde  de 
1’equipement  lors  de  demarrages  aux  basses 
temperatures. 


LCD  55  -  ARCHITECTURE  MATERIELLE 


CARTE  MERE 


PANNEAU  LCD 
EQUIPS 
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.  Resolution  de  416  x  416  pixels  couleurs  au 
pas  de  310  microns 

Un  pixel  couleur  est  constitue  d'un  quadruplet 
R,V,V,B  au  pas  de  155  microns. 

.  Technologie  militaire  k  matrice  active  (TFT 
en  silicium  amorphe)  et  Crystal  Liquide 
ndmatique  en  heiice. 

.  Lampes  It  cathodes  chaudes  permettant  un 
haut  rendement  lumineux  (>601umen/watt). 

.  Luminance  du  blanc  >  470  cd/m2  en  sortie 
de  produit. 

.  Contraste  sous  fort  dclairement  >5:1 
(lOOOOOlux  k  45°). 

.  Compatibility  JVN 
->  L’dlectronique : 

.  Generation  graphique : 

Developpement  de  circuits  VLSI  specifiques  : 
Processeur  graphique  et  processeur  de  lissage 
specialises  It  la  technologie  LCD  (techno  1 
micron,  35000  portes). 

.  Circuits  'DRIVERS"  specifiques  k  la 
commande  de  recran  LCD. 

->  L’ecran  Uctile : 

Cet  ecran  remplace  le  traditionel  poste  de 
commande  It  touches  mecaniques. 

->  Mdcanique  : 

Dimension  hors  tout  (LxPxH) 
367x168x210 
.  Volume  9  litres 
.  Poids  7  kilogrammes 

3.  PERFORMANCES  DES  CRISTAUX  UQUIDES  EN 
VISION  DIRECTE 

Nous  allons  dans  ce  paragraphe  detailler  les 
performances  les  plus  remarquables  de  cette 
technologie,  celles  qui  justifient  l’engouement 
actuel.  Nous  nous  sommes  bases  pour  ce  faire 
sur  (’experience  de  Sextant  Avionique  dans  ce 
domaine  et  notamment  sur  les  travaux 
effectues  dans  le  cadre  des  programmes  Rafale 
et  Tigre. 

Encombrement/ poids  : 

Les  dcrans  k  cristaux  liquides  font  partie  de  la 
famille  des  ecrans  plats  et  permettent  une 
reduction  considerable  du  volume  et  du  poids 
en  comparaison  avec  les  tubes  k  rayons 
cathodiques. 


La  profondeur  due  k  1’ecran  est  de  l’ordre  de 
quelques  centimetres  auquel  il  faut  ajouter  la 
source  lumineuse,  suivant  les  performances 
requises.  Un  moniteur  peut  n’avoir  que  10  k 
20  centimetres  de  profondeur  k  comparer  avec 
les  trente  k  quarante  centimetres  d’une 
visualisation  k  tube  (avionique  complet  avec 
alimentation,  protections,  connecteurs,...). 

Lisibilite  sous  fort  6clairement : 

C’est  ici  un  des  points  majors  de  cette 
technologie  et  qui  mdrite  de  s’attarder.  Comme 
il  a  ete  indique  dans  le  paragraphe  precedent, 
les  dcrans  k  cristaux  liquides  ne  sont  pas 
emissifs  mais  transmissifs  ;  aussi  les  2  critdres 
essentiels  permettant  de  mesurer  leur  quality 
sous  fort  edairement  sont : 

-  la  transmission, 

-  la  reflectivity  spdculaire  et  la  reflectivity 
diffuse. 

La  transmission  varie  suivant  les  foumisseurs 
et  la  technologie  de  3  k  5  %  pour  un  ecran 
couleur.  Ce  qui  permet,  suivant  l’eclairage 
arrikre  d’obtenir  une  luminosity  depassant  les 
700  cd/m2. 

La  reflectivity  (speculate  ou  diffuse)  des 
ecrans  LCD  peut  fttre  extrftmement  faible  de 
1’ordre  de  0.8%  en  speculate  (voire  inferieure 
k0.5%)etde  0.15  %  endifius  (jusqu’k  0. 1  %) 
pour  peu  que  les  materiaux  constituants  l’ecran 
aient  ete  choisis  en  tenant  compte  de  ce 
critfere. 

Cette  caracteristique  essentielle  se  traduit  par 
la  possibility  de  lire  facilement  les 
informations  affichdes  (video  alphanumerique 
et  graphique)  quelque  soit  les  conditions  de 
luminosity  exterieure.  Des  contrastes  ratios 
supdrieur  k  5  ont  ete  obtenus  avec  cette 
technologie;  mesures  effectudes  dans  les 
conditions  de  la  norme  MI1-L-85762A  (10000 
fc,  2000  fl). 

Disponibilite/Fiabilite  : 

Nous  n’avons  pas  encore  assez  de  recul  dans 
le  domaine  des  matrices  actives  pour  fitre 
catdgorique  sur  ce  point.  Cependant 
nous  allons  developper  3  iddes  qui  nous  font 
penser  que  Ik  encore  cette  technologie  est 
interessante. 

Les  cristaux  liquides  non  actifs  (dedies  ou 
multiplexes)  utilises  depuis  plus  de  quinze  ans 
ont  montre  une  fiabilitd  extrSmeme  it  dlevde. 
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Les  dcrans  2k  matrice  active  sont  constituds  de 
millions  de  transistors  ou  diodes.  La  perte 
d’un  de  ces  Elements  ne  remet  pas  en  cause  la 
quality  de  l’information  transmise.  Nous  dirons 
que  ces  dcrans  sont  toldrants  aux  pannes. 

Les  alimentations  ndcessaires  au 
fonctionnement  de  la  visualisation  sont  de  type 
basse  ou  moyenne  tensions  (y  compris  la  boite 
2k  lumidre)  et  non  pas  THT  comme  les  CRT. 

Qualitd  d’image/Prdcision  : 

Les  AMLCD  offrent  ce  double  avantage  de  la 
numdrisation  (matrice)  et  de  I’efifet  optique 
analogique.  La  transmission  de  l’information 
du  capteur  jusqu’au  pilote  pcut  Stre 
entidrement  numdrique  prdservant  son  intdgritd 
et  sa  qualitd,  plus  de  jitter  ni  de  bruit. 

Electron! que  : 

Les  LCD  ndcessitent  de  nouveiles  functions 
dlectroniques  : 
boite  2k  lumi&re, 
gestion  dcran  LCD, 

dvolution  des  fonctions  tels  le  traitement  viddo 
et  graphique. 

Mais,  avec  la  suppression  de  la  THT,  le  bilan 
en  terme  de  compldxitd,  encombrement  et 
consommation  est  largement  positif. 

Futur  ddveloppement  : 

Ce  tour  d’horizon  des  performances  des 
AMLCD  ne  serait  pas  complet  si  nous 
n’dvoquions  pas  quelques  domaines  ou  des 
amdliorations  sont  i  trouver.  En  premier  lieu 
Tangle  de  vue  important  deraandd  par  les 
avions  de  transports  ou  certains  hdlicoptbres; 
de  gros  efforts  techniques  et  financiers  sont 
effectuds  qui  doivent  permettre  de  trouver  le 
composant  optique  qui  augmentera  le 
contraste. 

Les  dcrans  LCD  deviennent  plus  grands 
(jusqu’k  14”  de  diagonale),  les  caractdristiques 
gdndrales  sont  relativement  constantes 
(rdsolution,  transmission,  architecture 
dlectronique). 

Si,  dans  le  domaine  de  Tangle  de  vue,  un 
effort  doit  fttre  fait  pour  permettre  une  vision 
pilote/copilote,  en  contrepoint  le  concepteur  de 
cockpits  se  doit  de  prendre  en  compte  cette 
caractdristique  des  dcrans  LCD  dds  la  crdation 
de  la  olanche  de  bord. 


4.  LES  VISUALISATIONS  EN  PROJECTION  : 

La  technologie  LCD  ne  semble  pas  pour 
Tinstant  compatible  de  trds  grand  format  mais 
de  trds  haute  ddfinition.  Dans  la  course  pour  le 
marchd  de  la  tdldvision  haute  ddfinition,  de 
gros  investissements  sont  faits  pour  le 
ddveloppement  de  valve  &  cristaux  liquides  de 
rdsolution  descendant  jusqu’k  40  microns  de 
pas  et  moins,  permettant  la  fabrication  de 
TVHD. 

Une  dvolution  similaire  nous  semble  pouvoir 
se  faire  dans  la  visualisation  avionique,  non 
seulement  pour  rdaliser  des  moniteurs  haute 
ddfinition  ayant  des  performances  de  volume 
de  poids  et  de  consommation  meilleures  que 
celles  des  TRC,  mais  aussi  pour  rdaliser  des 
visualisations'  colHmatdes  (visualisation  tftte 
moyenne). 

Ces  visualisations  collimatdes  offrent  2 
avantages  essentiels: 

-  Collimatde  k  Tinfmi,  1’information  affichde 
sur  Tdcran  semble  en  continuitd  avec  le  viseur 
tdte  haute  conduisant  k  un  accds  rapide  des 
donndes  par  le  pilote. 

-  La  densitd  d’informations  prdsentables  sur 
une  telle  visualisation  n’est  pas  proportionnelle 
2k  la  surface  de  Tdcran  mais  au  champ  de  vue. 

4.1  Visualisation  ttte  moyenne  : 

Nous  alions  rapidement  prdsenter  ce  que  peut 
fttre  une  telle  visualisation  se  basant  sur  une 
maquette  rdalisde  dans  le  cadre  d’un  marchd 
d’dtude,  et  en  ddtailler  les  performances. 

Les  caractdristiques  gdndrales  de  cette 
maquette  sont : 

-  Champ  d’une  vingtaine  de  degrds 

-  Ca*.  citd  couleur  (RGB  /  >  16  niveaux  de 
gris) 

-  Rdsolution  nombre  de  points  >  230.000 

Architecture  Optique  : 

La  figure  4.1.1  montre  (’architecture  optique 
comportant  des  dldments  holographiques. 

Cette  architecture  utilise  comme  source 
d’image  3  LCD  superposds  et  en  projection. 
Compldtde  par  une  optique  de  collimation,  elle 
rdpond  aux  contraintes  avioniques  : 

-  la  lampe  est  de  type  halogbne  (s.  nplicitd, 
taille  rdduite...). 
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-  une  deuxidme  lampe  a  dtd  rajoutde  afm 
d’ augment er  la  sdcuritd  du  systdme, 

-  l’utilisation  de  composants  holographiques  a 
permis  une  reduction  du  volume  total. 

-  une  compensation  colorimdtrique  est 
possible: 

•  elle  permet  d’dviter  une  derive  des  couleurs 
dans  toute  la  gamme  d ’utilisation  (jour/nuit) 

•  1’optique  de  projection  a  un  encombrement 
rdduit  (1  <  140mm)  et  utilise  des  surfaces 
complexes  (asphdriques...). 


1.  Lampes 

2.  S^paraleur  de  couleurs 

3.  Mflangeur 


4.  Optique  de  projection 

5.  Diffuseur 

6.  Optique  de  collimation 


Figure  4.1.1  Architecture  optique 


Architecture  dlectronique 
Elle  est  composde  : 

-  d’un  traitement  des  donndes  viddo  pour 
s’adapter  parfaitement  aux  caractdristiques 
optiques  et  dlectrooptiques  des  LCD, 

-  d’un  systdme  de  gestion  qui  prend  en  compte 
I’environnement  intdrieur  et  extdrieur  de  la 
visualisation  et  modifie  en  consequence  le 
traitement  viddo. 

Performances 

-  Champ  : 

.  total :  20°  x  20° 

.  instantand  :  20°  x  20° 

.  Cet  dcran  occupe  le  champ  visuel  equivalent 
4  celui  qu’occuperait  un  dcran  de  30 
centimetres  de  c6td  (12"xl2")  placd  it  80 
centimetres  de  l’oeil  du  pilote. 

-  Luminance  : 

.  R  (605-640  nm)  >  150  cd/m2 
.  V  (500-590  nm)  >  400  cd/m2 
.  B  (440-460  nm)  >  80  cd/m2 
.  blanc  >  600  cd/m2 

-  Contraste  (sous  un  dclairement  de  lOOOOOlux 
4  45°  d’inclinaison)  : 


.  R  >  4:1 
.  V  >  10:1 
.  B  >  3:1 


-  Colorimdtrie  : 

.  La  figure  suivante  montre  le  diagramme  de 
couleur. 


Figure  4.1.2  :  Diagramme  des  couleurs 


4.2  Visualisation  de  type  rdtro  projecteur  : 

Ces  visualisations  se  prdsentent  comme  des 
moniteurs  en  vision  directe  mais  le  principe  de 
gdndration  d’image  est  celui  d’un  projecteur. 

L’intdrdt  majeur  de  ce  type  de  visualisation  se 
situe  dans  le  domaine  des  grands  dcrans  de 
forte  resolution  (Cartographic,  Radar...),  d’un 
point  de  vue  technique  les  dcrans  LCD  sont 
des  composants  optiques,  il  faut : 
utiliser  leurs  caractdristiques  sans  contraintes 
d’angle  de  vue, 

ddgager  la  gdndration  d’images  des  contraintes 
cockpits  (un  seul  type  d’dcran  LCD  pour 
diffdrents  moniteurs). 

L’architecture  d’ensemble  est  identique  4  celle 
des  visualisations  tdtes  moyennes,  seule  la 
demidre  couche  optique,  la  collimation,  est 
supprimde.  II  est  de  plus  ndcessaire  d’agrandir 
l’image  (voir  figure  4.2.1). 

Performances  objectives  : 

-  Definition  :  jusqu’4  1000x1000  points 
couleurs 

-  Luminositd  :  200  cd/m2  aujourd’hui  jusqu’4 
400  cd/m2 

-  Contraste  sous  100000  lux  suivant  la 
luminositd  de  2:1  4  4:1. 

Analyse  critique 

La  projection  4  base  d’dcrans  LCD  est  une  des 
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voies  qui  permettront  de  rdaliser  des 
televisions  haute  definition  (TVHD).  De  gros 
investissements  sont  consentis  dans  ce  domaine 
pour  la  realisation  de  valves  au  format  4/3  ou 
16/9  ayant  un  pas  de  l’ordre  de  40  It  60/*. 

La  projection  permet  la  conception  d’un  coeur 
"generation  d’ images"  independamment  de 
l’ecran  qui  sera  presente  k  l’utilisateur,  d’oCt  la 
possibilite  de  concevoir  des  visualisations 
grand  format  et  haute  resolution  adaptees  au 
besoin  des  utilisateurs  k  partir  de  matrice 
standard. 

L’industrialisation  de  ce  type  de  produit 
permettra  de  compenser  la  complexite 
inherente  k  ce  systfcme.  Cependant,  ces 
visualisations  seront  plus  plus  complexes  que 
les  visualisations  LCD  en  vision  directe. 


1.  Lampes  4.  Optique  de  projection 

2.  Sdparateur  de  couleurs  S.  Diffuseur 

3.  Mflangeur 

Figure  4.2. 1  Architectui  i  optique 


5.  LES  COCKPITS  LCD 

La  conception  des  cockpits  a  suivi  revolution 
technologique.  Les  ecrans  k  tubes  k  rayons 
cathodiques  ont  remplace  les  visualisations 
electromecaniques,  permettant  par  la  suite  de 
regrouper  les  informations  sur  des  ecrans  de 
grande  taille. 


Les  TRC  sont  aujourd’hui  remplacds  par  les 
LCD.  Nous  nous  proposons  de  faire  un  rapide 
survol  des  potentialites  que  cette  nouvelle 
technologie  offre  aux  concepteurs  de  cockpits. 
Les  figures  S.l,  5.2  montrent  revolution  du 
cockpit  de  Fheiicoptfere  SUPER  PUMA  et  la 
figure  5.3  est  une  illustration  du  cockpit  de 
I’ avion  Rafale  D  k  base  de  LCD  en  vision 
directe  ou  en  projection. 

De  nouveaux  concepts  de  visualisation 
apparaissent  type  notebook,  cartographie  ou 
bibliothfeque  eiectronique.  Le  faible 
encombrement  des  ecrans  k  cristaux  liquides, 
leur  resolution  et  la  qualite  d’image,  associes 
k  une  consommation  reduite  prennent  tout  leur 
interfit. 

A  plus  long  terme,  les  besoins  des  utilisateurs 
nous  semblent  doubles  : 

-  plus  d’ informations  de  meilleure  qualite, 

•  un  accfes  facile  k  ces  informations 
(ergonomie). 

La  technologie  LCD  permettent  une  avancee 
dans  ces  deux  domaines  : 

-  haute  resolution  et  niveaux  de  gris  permettent 
qualite  d’image  et  densite  d’ informations, 

-  ecran  numdrique,  les  LCD  s’integrent 
parfaitement  dans  les  chaines  de  traitement  du 
signal  en  conservant  la  qualite  d’image  (zoom, 
translation,  mddaillonage,  regie  video,  ...), 

-  visible  quelle  que  soit  1’ambiance  lumineuse, 
le  pilote  a  toujours  accfes  k  l’information, 

-  en  projection  P ecran  s’adapte  aux  besoins  en 
terme  de  geometric  et  d’ergonomie. 
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Discussion 


QUESTION  J.  DANSAC 

Pensez  vous  que  la  technologie  LCD  pourra  etie  appliquee 
dans  le  futur  aux  HUD  ou  HMD? 

REPLY 

Cette  technologie  n'est  pas  immediateinent  adaptable:  ll 
faudrait  passer  de  dimensions  de  pixels  de  40 urn  a  des  pixels 
de  15  a  2  5  urn. 
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ABSTRACT 

The  Navy  is  exploring  the  use  of  an  adaptive 
autonomous  target  cuer  that  is  potentially  more 
reliable  than  existing  systems.  Variations  in  the 
operational  environment  and  target  type  are  major 
factors  that  can  degrade  an  autonomous  sensor-based 
cuer  performance.  The  solution  presented  here  is  to 
have  several  parallel  algorithms  for  each  functional 
component  in  our  target  cuer.  Each  algorithm  is  tuned 
to  handle  a  particular  situation.  Based  on  certain 
indicator  functions,  a  fuzzy  logic  expert  system 
controller  will  select  the  most  suitable  algorithm  and 
optimal  parameters  to  process  the  sensor  data. 

Although  the  proposed  system  is  essentially 
comprised  of  conventional  components  such  as  image 
preprocessing,  feature  extraction,  and  correlation,  it 
distinguishes  itself  because  of  its  use  of  fuzzy  logic  in 
decision  making  and  its  adaptive  nature.  In  designing 
an  expert  system,  we  will  select  the  optimal  set  of 
indicator  functions  using  the  Taguchi  process-control 
methodology.  Preliminary  results  will  be  presented. 

INTRODUCTION 

An  autonomous  target  cuer  on  a  land  attack  aircraft 
will  enable  the  pilot  to  launch  the  weapons  at  a 
greater  range  with  confidence,  reduce  the  time  of 
exposure  of  the  aircraft  to  threats,  and  improve  the 
survivability  and  effectiveness  of  the  aircraft. 
Improved  target  cuer  performance,  namely  higher 
target-detection  rate  and  lower  false-alarm  rate,  will 
enhance  the  acceptance  of  such  a  system  and 
encourage  its  use  by  the  flight  crew. 

However,  the  current  generation  of  target  cuers  has 
not  gained  wide  acceptance.  These  systems  perform 


satisfactorily  under  the  controlled  or  tested  situation, 
but  variations  in  operational  environments  adversely 
affect  system  performance  and  render  them  unreliable. 
Our  solution  is  to  build  a  cuer  that  has  switchable 
modules,  with  each  individually  tuned  to  operate 
optimally  under  a  specific  condition.  This  is  in  line 
with  the  modern  trend  of  machine  intelligence  which 
is  to  build  highly  specialized  expert  systems  (Ref  1). 
The  proposed  cuer  is  essentially  several  narrowly 
focused  cuers  tuned  to  operate  under  prescribed 
conditions.  A  controller  will  select  which  special 
purpose  cuer  to  use  based  on  the  input  from  on-board 
sensors,  from  the  pilot  or  operator,  and  from  the 
sensed  image  itself. 

CONTENTS  AND  ORGANIZATION 
In  the  next  section,  a  brief  review  of  previous  work  in 
this  area  and  their  relationship  to  the  present  study  is 
given.  Then,  we  will  discuss  a  system  overview  of  a 
generic  adaptive  autonomous  cuer.  Next,  the  structure 
of  the  fuzzy  logic  controller  is  explored  with  the 
introduction  of  a  step-by-step  recipe  for  selecting 
fuzzy  inputs  and  rules.  Lastly,  we  will  set  up  a  simple 
controller  that  will  select  an  appropriate  algorithm  to 
perform  edge  extraction  on  a  two-dimensional 
intensity  image. 

PREVIOUS  WORK  AND  THE  PROPOSED 
APPROACH 

Many  studies  on  adaptive  image  processing  are 
reported  in  the  literature  (see  Ref  2,  3,  and  4). 
However,  these  studies  focused  on  adapting  filter 
parameters  to  varying  local  statistics  within  a  given 
non-stationary  image. 

Previous  work  that  resemble  the  approach  taken  here 
has  not  been  widely  publicized.  Of  the  few  relevant 
reports  found,  the  adaptive  nature  of  the  system  was 
only  implied  and  not  explicitly  explained  or  stated. 
For  instance,  Ref  5  discusses  a  missile  defense  system 
that  would  select  in  real-time  various  image 
processing  filters  based  on  the  nature  of  the  input 
image.  However,  the  algorithms  selection  methods 
were  not  discussed. 

The  approach  described  in  this  paper  is  unique  in  two 
respects.  First,  the  controller  selects  the  appropriate 
filter  and  parameters  based  on  the  nature  of  the  sensed 
input  image,  such  as  image  statistics,  and  the 
operational  environment.  Second,  the  algorithm  and 
parameters  to  be  used  are  determined  by 
experimentation  with  both  real  and  synthetic  images. 

A  fuzzy  logic  expert  system  shell  was  chosen  to 
develop  the  controller  because  of  the  existence  of  fast 
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fuzzy  logic  hardware.  It  is  more  human-like  in 
decision  making  and  the  software  is  easy  to  use.  We 
use  the  Taguchi  method  to  select  the  critical  input 
parameters  to  the  fuzzy  logic  controller  and  to  set  up 
the  fuzzy  rules  because  it  is  simple  to  implement  and 
is  less  time  consuming  than  an  exhaustive  parameter 
study.  A  brief  introduction  to  fuzzy  logic  and  the 
Taguchi  method  is  presented  in  Annexes  A  and  B, 
respectively. 

COMPONENTS  OF  THE  ADAPTIVE 
AUTONOMOUS  CUER 

The  adaptive  autonomous  cuer  is  divided  into  several 
structural  components  as  shown  in  Fig  1. 

The  preprocessing  step  reduces  noise  in  the  image. 
Optionally,  the  volume  of  data  can  be  reduced  by 
using  a  region-of-interest  locator  and  processing  only 
this  sub-image  region. 

The  feature  extraction  module  further  reduces  the  data 
volume  and  transforms  the  pixel-space  image  to  a 
feature-space  representation.  A  good  feature 
representation  simplifies  the  model  correlation  or 
matching  task.  Some  examples  of  features  are  edges, 
corners,  area,  volume,  and  reflectivity  contrast.  The 
output  of  this  module  is  a  feature-space  map  of  the 
original  image. 

The  features  obtained  from  the  image  are  then  matched 
in  a  correlator  with  those  generated  by  the  target 
model  through  the  simulated  seeker  data  path  (shaded 


area  in  Fig  1).  The  output  of  the  module  is  a 
probability  of  target  presence. 

OVERVIEW  OF  FUZZY  LOGIC 
CONTROLLER 

Fig  2a  and  2b  shows  a  block  diagram  of  a  fuzzy  logic 
controller  for  adaptive  image  processing  and  the 
incorporated  fuzzy  logic  expert  system,  respectively. 
The  evaluation  box  on  the  left  (Fig  2a)  calculates 
parameters  that  will  indicate  the  type  and  quality  of 
the  image.  Some  examples  are  mean,  variance, 
average  gradient,  and  edge  density.  These  indicator 
functions  and  information  about  the  operational 
environment  are  input  to  a  fuzzy  logic  expert  system 
that  will  decide  which  filter  and  parameters  to  use  in 
processing  the  image. 

The  rules  in  the  fuzzy  logic  expert  system  are  initially 
set  up  using  hueristics  and  intuition  with  help  from  a 
human  expert.  Refinement  and  addition  of  new  rules 
are  achieved  through  experimentation  with  real  and 
synthetic  images.  The  procedure  is  outlined  below. 

1.  Establish  a  uniform  performance  measure  for  the 
filter.  For  preprocessing,  examples  are  signal-to- 
noise  ratio  or  percentage  of  speckles  removed.  For 
edge  extraction,  the  following  formula  is  appropriate. 


max(N;,Nd)  1  +  ad? 


OUTPUT 

TARGET 

REPORT 


Fig  1 .  Structural  Components  of  an  Adaptive  Autonomous  Cuer. 
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where  dj  is  the  distance  between  a  pixel  detected  as 
edge  and  the  nearest  predetermined  ideal  edge  pixel;  a 
is  an  adjustable  constant;  and  N;  and  Nd  are  the  total 
number  of  expected  edge  pixels  and  detected  pixels, 
respectively. 

2.  Perform  a  Taguchi  experiment  to  determine  under 
what  conditions  a  particular  filter  will  work  best.  For 
example,  assign  high,  medium,  and  low  values  to  the 
mean,  variance,  average  gradient,  range,  visibility, 
and  target  size  for  the  experiment.  The  outcome  of  the 
experiment  will  show  which  input  parameters  are 
important  in  determining  which  filter  to  use;  and  for 
the  particular  filter,  conditions  under  which  it  will 
perform  optimally.  In  other  words,  refine  the  fuzzy 
rules. 

3.  Repeat  step  2  for  each  filter. 

4.  Remove  those  indicators  or  inputs  that  have  the 
least  effect  on  the  performance  of  the  system  based  on 
the  results  of  the  Taguchi  experiments. 

In  the  next  section,  we  will  show  an  example  of  the 
initial  setup  of  a  fuzzy  logic  controller  to  select  an 
edge  extraction  filter. 

EXAMPLE  OF  A  FUZZY  LOGIC 
CONTROLLER 

For  extracting  edges  from  an  intensity  image,  we 
know  that  a  Sobel  operator  generally  works  quite  well 
(Ref  6).  However,  a  difference-of-Gaussian  operator  is 
better  if  the  image  feature  contrast  is  flat  or  the 
contrast  gradient  transition  is  less  abrupt,  and  a 
stochastic  filter  is  preferred  for  very  noisy  image 
features.  Based  on  this  a  priori  information,  a  gradient 
selection  membership  function  as  shown  in  Fig  3  and 
a  rules  matrix  for  the  controller  as  shown  in  Fig  4  can 
be  set  up. 


A  procedure  was  written  to  implement  this  fuzzy 
selection  algorithm.  The  fuzzy  control  surface  for  the 
edge  extraction  filter  is  shown  in  Fig  S. 

REMARKS  AND  CONCLUSIONS 
An  enabling  factor  for  this  system  is  the  availability 
of  faster  and  smaller  computer  hardware.  Fig  6 
illustrates  the  exponential  increase  in  the  processing 
power  of  large  computer  systems  over  the  recent 
decades.  This  tremendous  increase  in  processing 
power  is  achieved  via  faster  clock  speed,  parallel 
processing  on  multiple  CPUs,  and  smaller  chip  size. 
It  is  not  unreasonable  to  assume  that  a  teraflop 
computer  that  is  small  enough  to  be  put  on  an  aircraft 
could  be  available  before  the  end  of  the  century. 

Another  hardware  solution  is  the  use  of  special 
purpose  chips.  For  example,  the  fuzzy  logic 
controller  can  be  mapped  to  special  purpose  existing 
fuzzy  l.~<gic  chip  sets  that  can  process  200,000  rules 
per  second.  Alternatively,  the  fuzzy  rules  can  be 
mapped  to  analog  artificial  neural  network  hardware 
such  as  Intel's  ETANN™  chip  (Ref  7),  which  will  only 
take  1  or  2  clock  cycles  to  reach  a  result. 

RESULTS 

The  proposed  system  is  very  flexible  due  to  its 
interchangeable  parts;  and  a  new  methodology,  the 
Taguchi  method,  has  been  introduced  to  embed 
knowledge  into  an  intelligent  system.  This  system  is 
a  bold  attempt  to  solve  the  nagging  problem  of 
system  deterioration  due  to  varying  operational 
environments.  If  successful,  this  proposed  system 
will  perform  equally  well  in  the  sands  of  Saudi  Arabia 
or  in  the  jungles  of  South  America;  in  the  middle  of 
the  day  or  in  the  dead  of  night;  on  a  calm  sunny  day  or 
in  a  driving  rain  storm.  Such  flexibility  and 
robustness  will  go  a  long  way  in  increasing 
acceptance  of  autonomous  cueing  systems. 


FUZZY  SET:  GRADIENT  SELECTION 


Fig  3.  Stage  2  Gradient  Selection  Membership  Function. 
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Fig  6.  Processing  Power  of  Computers  (Computer  Speed  Versus  Year). 
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ANNEX  A:  A  BRIEF  INTRODUCTION 
TO  FUZZY  LOGIC 

The  objective  of  a  fuzzy  logic  system  is  to  use 
imprecise  knowledge  and  rules  in  a  particular  situation 
to  arrive  at  a  particular  conclusion  or  decision  and  to 
represent  this  in  a  modular  structure.  To  accomplish 
this,  however,  it  is  necessary  to  address  the  issue  of 
uncertainty.  This  issue  is  critical  because  much  of  the 
information  in  the  knowledge  base  is  imprecise  or 
incomplete.  Fuzzy  logic  is  a  formulation  of  logic  that 
deals  with  such  inherently  fuzzy  human  concepts  as 
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low,  very,  most,  etc.,  in  a  rigorous  mathematical  fuzzy  subsets  membership  function.  For  example,  we 
framework.  might  define  the  fuzzy  subset  'LOW  noise'  by  Fig  A.l. 


In  conventional  set  theory,  a  particular  object  is 
either  a  member  of  a  given  set  or  is  not.  In  fuzzy 
subset  theory,  a  particular  object  has  a  degree  of 
membership  in  a  given  set  that  may  be  anywhere  in 
the  range  0  (not  in  the  set)  to  1  (completely  in  the 
set).  While  Boolean  logic  would  require  a  statement  to 
be  true  or  false,  fuzzy  logic  would  assign  a  degree  of 
membership  to  the  set. 

Fuzzy  logic  also  defines  operators  on  degree  of 
membership  values:  AND,  OR,  and  NOT. 

Operator  Mfianifl* 


Knowledge  is  specified  to  the  expert  system  in  the 
terms  of  production  rules.  For  example: 

RULE  000 

IF  noise  is  LOW  AND  transition  rate  is  LOW  THEN 
gradient  is  LOW 

When  the  data  value  is  fuzzy,  the  degree  of 
membership  is  determined  as  the  maximum  degree  of 
membership  value  for  the  intersection  of  the 
membership  function  for  the  fuzzy  data  value  and  the 
fuzzy  set  as  shown  in  Fig  A.2. 


aANDb 

the 

minimum  of  the 

degrees 

of 

membership  for  a  and  b 

aORb 

the 

maximum  of  the 

degrees 

of 

membership  for  a  and  b 


NOT  a  one  minus  the  degree  of  membership  for  a 

Fuzzy  subsets  differ  from  conventional  sets  in  that 
membership  in  a  fuzzy  subset  does  not  have  sharp 
boundaries.  The  mapping  between  the  elements  of  the 
universe  of  discourse  and  their  corresponding  degree 
of  membership  in  the  fuzzy  subset  is  referred  to  as  the 


The  process  of  applying  the  degree  of  membership 
computed  for  a  production  rule  to  the  rule's  conclusion 
to  determine  the  action  to  be  performed  is  called  an 
inference.  The  concept  is  that  the  value  to  be  assigned 
to  the  output  is  scaled  or  clipped  to  the  degree  of 
membership  for  the  premise,  and  that  all  of  the  scaled 
or  clipped  sets  for  all  of  the  rules  that  set  this  output 
are  unioned  together  to  form  the  final  output 
membership  function.  The  inference  method  results  in 
fuzzy  values.  The  centroid  defuzzification  method 
picks  the  output  value  corresponding  to  the  centroid 
of  the  output  membership  function  as  the  crisp  and 
definitive  value  for  the  output. 


FUZZY  SET:  NOISE 


FtANGE  OF  MEASURE 
Fig  A.l.  Membership  Function  for  Noise. 
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FUZZY  SUBSET 


Fig  A.2.  Determination  of  Degree  of  Membership  Value. 


ANNEX  B:  A  BRIEF  INTRODUCTION 
TO  THE  TAGUCHI  METHOD 
The  Taguchi  method  is  a  robust  parameter 
optimization  and  quality  control  technique.  The 
overall  aim  of  all  quality  control  activities  is  to 
produce  a  product  that  is  robust  with  respect  to  all 
noise  factors  (uncontrollable  factors  such  as 
temperature,  humidity,  individual  operators,  etc.). 
Robustness  implies  that  the  product's  functional 
characteristic  is  least  sensitive  to  variations  that  arise 
from  the  noise  factors.  Parameter  design  examines 
interactions  between  control  factors  and  noise  factors 
in  order  to  achieve  robustness.  The  Taguchi  process  is 
a  search  for  parameter  levels  at  which  a  characteristic 
is  most  stable.  The  technique  of  defining  and 
investigating  all  possible  conditions  in  an 
experiment  involving  multiple  factors  is  known  as 
the  design  of  full  factorial  experiments.  The  Taguchi 
approach,  using  fractional  factorial  methods, 
investigates  only  a  portion  of  all  the  combinations  of 
production  variable  values  and  then  estimates  the 
specific  combination  of  variable  values  that  will 
optimize  the  desired  results.  The  Taguchi  method  can 
save  considerable  time  and  money  (as  opposed  to  the 
full  factorial  method)  in  determining  an  optimum 
combination  of  variable  values.  However,  the  Taguchi 
method  requires  an  investment  in  carefully  planning 
the  tests  that  are  to  be  run  and  carefully  adhering  to 
the  method  during  testing  and  analysis  in  order  to 
optimize  the  success  of  its  application.  A  special  set 
of  orthogonal  arrays  (OAs)  is  constructed  to  lay  out 


the  experiment.  The  OA  facilitates  the  experiment 
design  process.  To  design  an  experiment  is  to  select 
the  most  suitable  orthogonal  array,  assign  the  factors 
to  the  appropriate  columns,  and,  finally,  describe  the 
combinations  of  the  individual  experiments  (called 
the  trial  conditions). 

The  results  of  the  experiments  are  analyzed  to  achieve 
the  three  objectives: 

1.  To  establish  the  best  or  the  optimum  condition  for 
the  process. 

2.  To  estimate  the  contribution  of  individual  factors. 

3.  To  estimate  the  response  under  the  optimum 
conditions. 

The  technique  is  applied  in  four  steps: 

1.  Brainstorm  the  quality  characteristics  and  design 
parameters  important  to  the  process. 

2.  Design  and  conduct  the  experiments. 

3.  Analyze  the  results  to  determine  the  optimum 
conditions. 

4.  Run  a  confirmatory  test  using  the  optimum 
conditions. 


Discussion 

QUESTION  J.  DANSAC 

Quand  pensez  vous  que  ces  techniques  seront  adaptees  dans  des  missiles? 

REPLY 

In  about  twelve  years. 
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Equipment,  more  or  less  ready  to  be  used  in  Helicopters 

H.HeUmuth  and  Dr.H.-D.V.Bflhm 
Eurocopter  Deutschland  GmbH  Germany 
Avionics 

Postfach  80  11  40 
D-8000  Munchen  80 


SUMMARY 

Today  there  is  a  long  list  of  equipment  proposed  by 
the  Equipment  Industry  ready  to  be  used  in  a  heli¬ 
copter. 

Once  being  in  the  situation  to  define  system  concepts 
for  civil  and  military  helicopters  to  fulfill  a  specific 
mission  under  several  project  boundary  conditions,  a 
closer  look  at  these  equipment  will  shorten  the  list 
dramatically. 

Boundary  conditions  are  for  example  : 

-  low  price  /  low  live  cycle  costs 

-  commercially  available 

-  maintainability 

-  standard  equipment 

-  short  time  availability 

-  ergonomic  requirements 

-  flight  safety 

-  mission  effectiveness 

-  low  crew  workload 

-  vulnerability  /  redundancy 

-  integration  constraints 

There  are  technologies  proposed  on  the  market  since 
several  years  which  have  not  been  used  in  series 
because  of  ineffectiveness  or  simply  because  of  being 
not  ready  to  be  used. 

For  helicopters,  there  are  specific  requirements  which 
limit  the  use  of  some  equipment  feasible  for  fixed 
wings. 

This  paper  will  show  examples  of  equipment  taken 
from  project  studies,  and  show  the  specific  boundaries 
for  helicopters.  Avionic  equipment,  sensors  as  well  as 
displays  and  controls  will  be  covered.  The  paper  will 
analyze  equipment  and  technologies  ready  to  be  used 
today  and  in  near  future  as  well  as  equipment  hardly 
usable.  Equipment  specific  characteristics  and 
helicopter/cockpit  integration  specifics  will  be  cover¬ 
ed. 

1.  INTRODUCTION 

1 . 1  Helicopter  Flight  path 

In  contrast  to  the  fixed  wings,  a  helicopter  flies  lower. 
The  so  called  Nap  of  the  Earth  (NOE)  flying  is 
performed  in  heights  above  ground  between  3  and  SO 
feet  (  between  1  and  15  meters  )  at  a  speed  of  approx. 
0  to  40  knots  at  night  and  0  to  70  knots  in  daytime. 


This  NOE  flight  is  always  performed  by  looking 
outside,  heads  up.  Armed  or  ’naked*,  the  eyes  must 
look  out  of  the  cockpit  to  guide  the  helicopter,  to 
select  the  flight  path.  There  is  no  time  to  look  inside 
the  cockpit.  All  additional  information  coming  from 
sensors  and  systems  must  be  presented  head  up. 

This  helicopter  specific  condition  has  to  be  kept  in 
mind,  while  reading  this  paper! 


Fig.  1  NOE  Flight  Path 
(Source.  ATV  Buckeburg) 

1.2  Bottleneck  /  Electro-magnetic  Spectrum 
At  the  beginning  of  aviation,  the  information  was 
restricted  mainly  to  that  directly  coming  to  the  natural 
organs  of  sense  like  eye  and  sense  of  balance.  More 
and  more  artificial  sensors  are  being  used,  detecting 
signals  of  various  wavelength  which  are  translated  into 
the  area  of  natural  perception. 

These  sensors  are  beginning  with  very  simple  ones  like 
the  magnetic  compass  or  the  barometric  altimeter  and 
nowadays  ending  with  equipment  like  Infrared  and 
Micro-Wave  Radar  Sensors. 

The  Electro-magnetic  Radiation  Spectrum: 

Optical  radiation  is  defined  as  electro-magnetic 
radiation  in  the  range  of  wavelengths  between  lOnm 
and  1mm  and  can  be  subdivided  into  UV  (Ultraviolet), 
visible  radiation  (light)  and  1R  (infrared);  compare 
Fig. 3. 

In  order  for  a  pilot  to  be  able  to  see  adequately  at  night 
or  in  bad  weather,  be  must  be  provided  with  some 
artificial  means  of  extending  his  optical  range  to 
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counteract  the  effects  of  poor  light  or  atmospheric 
interference.  Either,  existing  electro-magnetic  energy 
levels  need  to  be  amplified  electronically,  or  some 
method  of  observation  must  be  used  at  wavelength 
which  can  effectively  disregard  atmospheric  absorption 
owing  to  mist  or  fog  particles.  Fortunately,  there  are  a 
number  of  narrow  bands  or  'windows*  in  the  visible 
and  IR  region  of  0.3pm  to  14pm  for  which 
atmospheric  absorption  is  minimal,  i.e.  transmittance 
is  maximal,  compare  fig. 3. 

The  human  eye  responds  only  to  optical  radiation 
with  wavelength  in  the  visible  region  of  the  electro¬ 
magnetic  radiation  spectrum  i.e.  from  0.4pm  to 
0.7pm.  Two  types  of  light-sensitive  cells  are  present  in 
the  retina  of  the  eye,  the  cones,  used  for  colour  vision 
in  day-light,  and  the  rods,  not  so  highly  selective  in 
colour  range,  but  most  responsive  to  low-intensity 
sources  i.e.  at  night. 

Visual  aids,  in  the  form  of  multispecial  electro-optic 
sensors,  can  be  divided  into  3  categories: 

-  Image  Intensifiers  (ID 

-  Thermal  Images  (TI) 

-  Image  Micro-wave  Radars  (in  near  future) 

Conspicuous  is  the  tact,  that  almost  all  information 
provided  by  sensors  and  systems  are  transferred 
visually  to  the  pilot. 


Fig.  2  Electromagnetic  Spectrum 
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Fig.  3  Atmospheric  Transmittance 

rhe  enormous  increase  of  data  due  to  the  new  sensors 
leads  to  a  strong  increase  of  pilots  workload.  The  limit 
of  faulty  data  processing  by  the  pilot  is  reached. 
Careful  selection  of  necessary  data  and  its  kind  of 
presentation  for  specif':  mission  situations  is  needed  to 
avoid  the  overload  of  the  crew. 

Just  as  the  cockpit  provides  information  about  systems 
and  environment  to  the  pilot,  it  receives  or¬ 
ders/commands/controls  from  the  pilot  purely  by 
manual  action. 

The  one-sided  use  of  human  organs  for  information 
recording  and  command  transfer  is  about  to  be  broken. 
Attention  !  Some  techniques  promise  too  much. 

2.  Visual  Aids  and  Mulispectral  Imaging  Sensors 
Image  Intensifier  (ID  tubes  as  they  are  well  known 
within  Night  Vision  Goggles  are  integrated  in  the 
helmet  together  with  micro-CRTs.  This  allows 
combination  of  II  pictures  and  symbology  and  platform 
mounted  sensors  with  symbology. 

Thermal  Images  (TI)  today  have  a  good  quality.  They 
are  used  in  the  3  to  5  and  8  to  12  pm  spectral  range  for 
day  and  night  application  on  a  steerable  platform  in  the 
helicopter.  The  steering  arrays  based  on  HgCdTe- 
detector  materials  with  approx.  400  000  pixels  per 
image  (video  standard)  are  not  available  today  but  the 
PSi-detector  technology  in  the  3  to  5  pm  band  can  be 
used  without  scanner.  It  is  not  necessary  to  cool  down 
this  detector  up  to  liquid  nitrogen  temperature.  A 
multi-stage  Peltier  cooling  unit  is  sufficient. 

Passive  millimeter  wave  imaging  radiometers  are  the 
future  sensors  to  penetrate  rain  and  fog  better  than  TI. 
These  sensors  arc  used  for  signature  research  in 
satellites  with  remote  sensing  and  on  ground  trails 
(ref.l  and  2  )  in  the  35GHz,  94GHz  and  140GHz 
range  with  GaAs  technology.  Active  radar  systems  e.g. 
SAR  (Synthetic  Aperture  Radar)  or  side  and  forward 
looking  radar  are  good  for  battlefield  surveillance. 
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Detection,  recognition,  identification  and  interpretation 
of  multi -spectral  visual  aid  information  is  the  future 
technology  called  sensor  or  image  fusion.  The  single 
components  of  such  technologies  are  ready  but  the  way 
to  a  lightweight  and  cost  effective  device  will  take 
some  years.  To  this  imaging  sensors  belong  platform 
tracker,  multi-target  tracker  and  videomemory.  A 
videomemory  can  store  images  of  a  landscape  view  for 
a  short  time.  The  pilot/copilot  is  than  able  to  replay  the 
scanned  area  behind  safe  obstacles. 

3.  Helmet  Sensors  and  Displays 
The  requirements  for  a  helicopter  with  night-flight 
capabilities  have  increased  in  the  last  years.  The 
reasons  are  the  experience  with  IIs  and  TIs  in  bad 
weather  conditions  in  the  European  area.  A  pilot 
visionics  system  (PVS)  consisting  of  a  steerable 
platform  with  electro-optical  sensors  is  controlled  by  a 
helmet-mounted  sight  (HMS),  also  called  tracking 
system.  The  purpose  of  the  HMS  is  to  steer  without  an 
additional  workload  either  a  sensor-platform,  a  landing 
light  platform  or  a  weapon  platform  in  accordance  with 
the  head  motion  of  e.g.  a  H/C  crew  .  The  measured 
values  of  head  motion  angles  must  be  of  high  accuracy 
and  to  be  available  with  a  minimum  of  time  delay. 

The  helmet-mounted  display  (HMD)  uses  one  or  two 
mini  cathode  ray  tubes  (CRT)  to  provide  a  video  image 
of  the  electro-optical  sensor  with  superimposed 
symbology.  With  the  help  of  a  HMS-fiinction  the 
image  can  be  roll-stabilized  to  keep  the  sensor  image 
horizon  aligned  with  the  natural  horizon  when  the  pilot 
moves  his  head  in  roll. 
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Modem  Integrated  Helmet  Systems  (IHS)  consist  of 
CRTs  for  displaying  binocular  images  of  TV-  or  TI- 
cameras  with  superimposed  flight  symbology,  see- 
through  capability  and  integrated  binocular  Image 
Intensifier  Tubes  (I IT),  compare  ref.  3  and  4. 

HT  with  implemented  CCD-Chip  will  be  the  future 
technology  for  the  image  fusion  with  TI  and/or 
imaging  MW-radar  (active  or  passive)  in  an  associated 
electronic  box.  The  IHS  represents  the  multispectral 
information  on  the  binocular  HMD  for  pilot  aids. 

4.  Obstacle  Warning  Systems 
Several  manufacturers  have  completed  basic  deve¬ 
lopment  and  first  operational  tests  of  Obstacle  Warning 
Systems.  Most  of  the  systems  are  based  on  active  milli¬ 
meter  radar  sensors  in  the  range  between  35  GHz  and 
94  GHz  but  also  on  laser  sensors  in  the  0.8  pm  or 
10  pm  spectral  range  (refs.5  to  8).  The  systems  in 
principle  are  functioning,  including  algorithms  to 
identify  the  obstacles  (in  the  most  cases  cables)  out  of  a 
cluttered  image. 

Constraints: 

-  the  sensor  package,  platform  is  very  bulky  and  heavy 

-  integration  of  the  sensor  on  a  helicopter  together  with 
optical  /  thermal  sensors  for  piloting  aids  and  target 
acquisition  and  weapons  firing  is  very  difficult,  becau¬ 
se  best  places  are  taken  by  the  other  sensors  already 

-  the  equipment  is  very  expensive,  development  costs 
will  have  to  be  paid  by  first  serial  integration. 

In  addition  to  all  the  above  mentioned  negative  facts, 
there  is  still  one  big  question  to  be  answered; 

"How  do  we  transfer  the  information  to  the  crew  ?’ 
Three  dimensional  means  of  presentation  for  distance, 
azimuth,  elevation/height  and  type  of  obstacle  is 
needed.  How  do  we  avoid  cluttering  of  display 
information?  How  can  we  display  only  that  informa¬ 
tion  which  is  not  already  presented  to  the  crew  by 
optical  means  like  Tl-pictures  etc.? 

We  do  not  know  the  answer  ! 


Fig.  4  Integrated  Helmet  System  (GEC) 


Fig. 5  Obstacle/Radar  Image  of  a  Valley 


Fig.6  Cockpit  View  of  the  Valley 


It  seems  to  us,  that  if  the  MMI  problem  can  be  solved 
(e.g.  on  an  IHS  with  stereoscopic  information 
presentation),  the  willingness  to  spend  more  money  on 
the  system  final  development  (miniaturization, 
serialization)  can  be  reached. 

Obstacle  Sensors  are  the  only  reliable  means  to  avoid 
undesired  contacts,  no  other  sensor  or  equipment  can 
replace  it ! 

5.  Electronic  Maps 

Electronic  Maps  are  a  perfect  help  to  unload  the  crews 
knees.  The  word  digital  means,  the  map  is  stored  as 
data  and  in  most  cases  in  a  constant  memory  storage. 
Memory  capacity  and  reading/processing  time  is  no 
longer  a  limit.  We  see  the  digital  map  as  a  absolute 
must  for  today's  and  future  military  helicopter. 

But  there  is  a  difference  between  on  the  market  availa¬ 
ble  digital  maps,  the  data  base. 

-  DMA-data  or  other  three  dimensional  data  is  availa¬ 
ble  to  present  a  terrain  model  in  3D-view  or  2D-view 
with  height  information  to  the  crew.  A  more  or  less 
artificial  image  of  the  rough  terrain  is  presented,  just 
as  if  taken  from  a  sensor. 


Fig.7  3D-View  DMA  Data 


Fig.  8  2D-View  DMA  Data 


-  The  other  technique  is  to  simply  digitize  paper-maps 
eventually  in  different  layers  (  layers  as  they  are 
needed  to  print  in  different  colors  by  adding  layer  for 

layer:  water,  railroads,  roads,  buildings  . ).  These 

layers  can  be  selected/combined  individually  to  the 
desired  combination  needed  to  fly  the  mission. 
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Fig.9  Scanned  Paper  Map 


At  a  first  look  the  3D  data  and  presentation  appears  to 
be  the  better  one,  but  what  does  a  low  flying  helicopter 
need? 

The  crew  wants  to  know  :  'When  I  turn  around  that 
comer  of  the  forest  to  my  right,  there  is  a  house  and  a 
low  electrical  power  line  ! ". 

This  information  today  can  not  be  given  by  a  DMA 
data  base  but  only  by  a  digitized  paper  map. 

Digitized  pixel  maps  appear  to  us  to  be  more  realisiic 
and  effective  during  the  mission  because  of  its  higher 
resolution.  Obstacle  warning  and  avoidance  sfill  can 
not  be  done  with  this  equipment,  but  obstacle  aware¬ 
ness  is  what  can  be  attained.  The  combination  of  map 
and  sensor  at  the  moment  promises  to  be  the  solution. 

6.  Voice  Command 

The  reliability  of  voice  recognition  still  is  not 
sufficient  to  perform  commands  with  any  safety 
criticality.  Mission  safety  relevant  functions  and  by  far 
flight  safety  relevant  functions  can  not  be  transferred 
by  this  technique.  What  remains  are  relatively 
unimportant  functions,  not  worth  to  spend  money  on. 

7.  Centralized  Controls 

Toady's  military  helicopters  and  those  of  tomorrow  are 
carrying  a  vast  number  of  equipment.  All  these 
equipment  have  to  be  controlled  and  monitored. 
Externally  the  cockpit  must  be  kept  as  small  as 
possible  at  least  for  better  external  vision.  Because  the 
space  needed  does  not  allow  individual  controls  and 
displays  -  leading  to  a  totally  cluttered  cockpit  - 
centralization  of  controls  and  displays  is  necessary. 
Most  of  the  actions  in  a  cockpit  are  done  to  select  what 
is  to  be  monitored  or  controlled  before  actually  doing 
it.  There  are  techniques  used  to  reduce  these  actions 
necessary  before  the  initial  task  like  automatic  or 
coupled  paging  and  moding,  HOCAS  (Hands  on 
Collective  and  Stick)  and  some  highly  frequented 


specific  controls  and  displays  (e.g.  RF1  -  Remote 
Frequency  Indicators). 

The  paging  tree  of  a  typical  Control  and  Display  Unit 
is  not  trivial. 

It  is  an  absolutely  necessary  to  have  a  clear  basic 
dialogue  throughout  all  pages  and  systems,  even  if  not 
optimized  for  some  of  those.  Paging,  selection, 
scrolling,  data  input  etc.  must  be  performed  in  one 
way  only. 

8.  Head  In  Display,  Cockpit  Integration 
A  totally  different  problem  is  the  physical  integration 
of  HID  type  (Head  In  Display)  of  oculars  into  a 
cockpit.  Originally  these  oculars  where  used  to  look 
through  sight  systems  via  direct  view  optics.  Today  the 
image  in  the  ocular  is  seen  on  a  high  resolution, 
monochrome  micro  display.  Up  to  now  only  these 
displays  provide  the  resolution  needed  for  target 
acquisition. 

The  head,  covered  by  the  helmet  is  trapped  between 
headrest  and  ocular.  When  turning  the  head  there  is 
contact  of  the  helmet  and  the  ocular.  Without  partial 
movement  of  the  ocular,  work  in  the  cockpit  is  more 
or  less  impossible. 

Work  is  continuing  to  delete  the  ocular  in  the  cockpit. 
Unfortunately  the  required  high  image  resolution  with 
equal  vision  angle  is  not  provided  by  Head  Down 
Displays  (HDD)  of  normal  size  in  the  surface  of  the 
glare  shield.  Color  displays  suffer  from  principally 
lower  image  resolution  over  monochrome  displays. 
Helmet  displays  with  mini  CRTs  do  not  have  the 
required  resolution. 


Fig.  10  Head  In  Display 
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9.  Global  Positioning  System  (GPS) 

Obviously  GPS  is  a  low  cost,  easy  to  handle  highly 
accurate  navigation  aid. 

Nevertheless  there  are  several  constraints  : 

-  Coverage;  still  there  are  holes  when  there  are  not 
enough  satellites  available! 

-  awareness  of  validity;  still  there  is  no  quick  (10  to  30 
seconds)  information  about  the  status  of  the  system 
provided,  so  you  can  not  relay  on  the  data,  even  if 
coverage  is  provided. 

-  only  with  additional  installations  (differential  GPS) 
will  accuracy  be  high  enough  for  precise  approaches 
and  obstacle  avoidance. 

-  There  are  nc  military  and  civil  ATC  procedures 
settled  today  to  perform  approaches  or  enroute  naviga¬ 
tion  based  on  GPS  alone. 

Because  of  being  very  cheap,  GPS  is  anyway  used  as  a 
"nice  to  have"  equipment,  but  never  rely  on  it! 
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Discussion 


QUESTION:  E.W.  PIJPERS 

In  your  paper  you  state  that  "electronic  maps"  would  help 
unload  the  pilot's  knee  (folders).  In  my  opinion  electronic 
maps  do  not  help  as  the  pilot  will  still  need  a  back-up  or  use 
the  knee  folder  as  a  secondary  source.  Please  comment. 

REPLY 


Back-up  Paper  Maps  are  still  needed,  but  only  in  case  of 
Electronic  Map  failure. 

During  normal  mission  there  is  no  map  in  the  cockpit  needed 
(paper  map) ,  these  maps  can  be  stowed  in  relatively  bad 
location 

QUESTION:  E.W.  PIJPERS 

In  your  paper  you  effectively  exclude  voice  command 
technology  for  near  term  use  in  helicopters.  I  have  to 
disagree  to  the  inference  made  that  there  are  no  other 
functions  being  either  safety  critical  or  unimportant .  Don't 
you  foresee  some  essential  functions  which  could  be  supported 
by  voice  command?  Furthermore  don't  you  feel  that  it  is 
frequently  the  normal  execution  of  non-essential  tasks  that 
interferes  with  the  execution  of  essential  and/or  critical 
tasks? 

Reliability  is  a  different  issue  from  the  error  occurring 
during  operation. 

On  principle  I  have  to  agree  to  your  statements. 

Nevertheless,  using  today's  and  tomorrow's  voice  command 
technologies,  whatever  function  you  would  like  to  perform,  the 
procedure  (e.g.  indicating  that  a  voice  command  is  coming, 
selection  of  the  function,  giving  the  command,  getting 
feedback,  confirming  this...)  will  increase  actions  and 
workload . 

In  case  a  critical  function  must  be  performed,  the  pilot  must 
concentrate  on  that,  any  other  command,  by  voice  or  manually, 
will  be  postponed,  until  the  critical  functions/commands  are 
completed . 
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